Augmented Reality Simulator

• US 20150260474A1
• Filed: 03/14/2014
• Published: 09/17/2015
• Est. Priority Date: 03/14/2014
• Status: Active Grant

• Current Assignee: Lineweight LLC
• Original Assignee: Lineweight LLC
• Inventors: Crye, Caleb Clark, Rublowsky, Stefan J.
• Granted Patent: US 9,677,840 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: F41A 33/00 Adaptations for training adaptations of barrels for recoil reinforcement F41A21/26; Gun simulators teaching or practice apparatus for gun-aiming or gun-laying F41G3/26 ; shooting games A63F9/02; military simulation G09B9/003
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G09B 9/003 for military purposes and tactics
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 3/2003 Display of colours specific for liquid crystal displays G09G3/3607

Abstract

An augmented reality system in which video imagery of a physical environment is combined with video images output by a game engine by the use of a traveling matte which identifies portions of the visible physical environment by techniques such as Computer vision or chroma keying and replaces them with the video images output by the video game engine. The composited imagery of the physical environment and the video game imagery is supplied to a trainee through a headmounted display screen. Additionally, peripheral vision is preserved either by providing complete binocular display to the limits of peripheral vision, or by providing a visual path to the peripheral vision which is matched in luminance to higher resolution augmented reality images provided by the binocular displays. A software/hardware element comprised of a server control station and a controller onboard the trainee performs the modeling, scenario generation, communications, tracking, and metric generation.

Claims

1. An augmented reality system, comprising:

a physical environment;

a headset for mounting to a trainee having eyes with a normal visual field;

binocular cameras mounted to the headset so as to supply a first source of binocular video feed of the physical environment, corresponding to a normal visual field of a trainee;

a binocular display mounted to the headset and connected to a second source of binocular video feed to provide binocular imagery to a trainee, wherein the second source of binocular video feed comprises a composited video combining the first source of binocular video feed of the physical environment with a third source of binocular feed comprising video game imagery which replaces portions of the first binocular video feed of the physical environment, so that the video game imagery and parts of the physical environment are displayed without substantial overlap;

a weapon simulator for carriage by a trainee, including a laser which projects a laser beam on to the physical environment.

2. The augmented reality system of claim 1 wherein the second source of binocular video feed is a compositor connected to the first source of binocular video feed and the third source of binocular video feed.

3. The augmented reality system of claim 2 further comprising a traveling matte processor arranged to process the first source of binocular video feed to extract portions of the first binocular video feed to generate a fourth source of traveling matte binocular video feed;

and wherein the traveling matte processor subtracts the fourth source of traveling matte video feed from a fifth source of binocular video feed of video game imagery written on a model of the physical environment to form the third source of binocular video feed.

4. The augmented reality system of claim 3 further comprising:

at least one tracking camera mounted to the headset;

wherein the physical environment has indicia written thereon and visible to the at least one tracking camera;

a processor hosting a CAD model of the physical environment in receiving relation to the at least one tracking camera, and arranged to produce a sixth source of video feed video as a view of the CAD model corresponding to the first source of binocular video feed of the physical environment;

a game engine producing a video game imagery and connected to the processor hosting a CAD model to write video game imagery to selected portions of the CAD model of the physical environment, the processor hosting the CAD model forming the fifth source of binocular video feed corresponding to the first source of binocular video feed of the physical environment and having the video game imagery written thereon; and

wherein the traveling matte processor is arranged to compare the first source of binocular video feed of the physical environment to the sixth source of video feed view of the CAD model to generate the traveling matte from portions of the physical environment which lack correspondence to the CAD model.

5. The augmented reality system of claim 4 wherein the processor hosting the CAD model is part of the game engine producing the video game imagery.

6. The augmented reality system of claim 4 wherein the traveling matte processor is arranged to compare the first source of binocular video feed to the sixth source of binocular video feed CAD model by a computer vision processor which draws an outline of the portions of the physical environment which lack correspondence to the CAD model and the outlines are filled into and generate the fourth source of traveling matte binocular video feed.

7. The augmented reality system of claim 1 wherein the binocular display mounted to the headset is arranged so as to provide a clear optical path to a trainee'"'"'s peripheral vision.

8. The augmented reality system of claim 7 further comprising smart glass arranged in the clear optical path to modulate a trainee'"'"'s peripheral vision to match a luminance provided by the binocular display.

9. The augmented reality system of claim 1 wherein the binocular display provides an image to a trainee'"'"'s peripheral vision.

10. The augmented reality system of claim 2 wherein the weapon simulator has portions marked with a visual signature for replacement in the compositor with images from a game engine.

11. An augmented reality system, comprising:

a real physical environment having an areal extent of at least about 40 ft.² and containing structures natural or artificial which form a set like a stage on which trainees may physically move by walking or running and moving through and around the structures to conduct training exercises;

wherein at least some of the structures of the physical environment have a multiplicity of indicia which are distinct one from the other;

a headset adapted to mount to the head of a trainee, such that the trainee wears the headset;

binocular cameras mounted to the headset so as to capture images of the physical environment corresponding to at least part of a trainee'"'"'s natural field of view and to output said images as a binocular video feed;

a headmounted display mounted to the headset and arranged for viewing by a trainee, so as to supply at least part of the trainee'"'"'s natural field of view;

at least one position tracking camera mounted on the headset and arranged to provide images of the indicia such that the position corresponding to the location of the headset is derived from said indicia;

a videogame engine which forms a source of video game imagery which augments the reality viewed by the binocular cameras as displayed on the headmounted display;

a compositor connected to the binocular video feed of the physical environment, and connected to the source of video game imagery;

a traveling matte processor arranged to process the binocular video feed of the physical environment to extract portions of the binocular video feed, and to generate a traveling matte therefrom;

wherein the traveling matte processor is connected to the compositor so that the traveling matte is subtracted from the video game imagery, and the game imagery is composited with the binocular video feed to form a source of composited video combining the binocular video feed of the physical environment and the game imagery; and

wherein the compositor is connected to the binocular display to display the binocular video feed augmented by the source of video game imagery.

12. The augmented reality system of claim 11 further comprising a processor hosting a CAD model of the physical environment, wherein the traveling matte processor is arranged to compare the binocular video feed of the physical environment to the CAD model and generate the traveling matte from an identification of portions of the physical environment which lack correspondence to the CAD model.

13. A method of conducting a real-time augmented reality training session, comprising the steps of:

marking portions of a physical environment;

imaging the physical environment with a device which captures a binocular video feed corresponding to the viewpoint of a trainee within the physical environment;

replacing the marked portions visible in the binocular video feed corresponding to the viewpoint of the trainee within the physical environment with images from a game engine to create a composited video feed showing portions of the physical environment and the images from a game engine;

conducting a training exercise in the physical environment which extends over space and time; and

during the training exercise observing the physical environment and the images from the game engine on a binocular display which shows without substantial overlap the composited visual feed of the physical environment and the images from the game engine.

14. The method of claim 13 further comprising the steps of:

creating a three-dimensional computer model of the physical environment, including the marked portions of the physical environment;

supplying video game imagery to fill at least one of said marked portions in the composited video feed;

processing the three-dimensional computer model and video game imagery so as to be aligned with the view angle of the device which generates the binocular video feed of the physical environment;

wherein the step of creating a composited video feed includes creating a correspondence between the three-dimensional computer model and portions of the physical environment contained in the binocular video feed;

wherein the compositing process further comprises;

identifying within the binocular video feed portions of the marked portions of the physical environment; and

creating a traveling matte corresponding to said identified portions and subtracting the traveling matte from the video feed to define subtracted portions and filling the subtracted portions with the video game imagery corresponding to the traveling matte, wherein the video game imagery is from the view angle of the binocular video feed.

15. The method of claim 14 wherein the step of marking portions of a physical environment is by chroma keying the portions of a physical environment to identify the marked portions.

16. The method of claim 14 wherein the step of creating the three-dimensional computer model of the physical environment includes the step of recording images of the physical environment by walking through the physical environment with the device which captures a binocular video feed, to form recorded images, and modifying the three-dimensional computer model so it better corresponds to the recorded images.

17. A method of conducting a real-time augmented reality training session for a trainee, comprising the steps of:

mounting a headset to a trainee'"'"'s head;

having the trainee enter a physical environment;

producing a first source of binocular video feed of the physical environment, corresponding at least partially to a normal visual field of the trainee from binocular cameras mounted to the headset; and

displaying on a binocular display mounted to the headset a second source of binocular video feed to provide binocular imagery to a trainee;

wherein the second source of binocular video feed is generated by compositing the first source of binocular video feed of the physical environment with a third source of binocular video feed comprising video game imagery which replaces portions of the first binocular video feed;

wherein the portions of the first binocular video feed replaced are identified by a signal in the first binocular video feed or by using machine vision comparisons of the first source of binocular video feed to the third source of binocular video feed, so that the video game imagery and parts of the physical environment are displayed to the trainee on the binocular display without substantial overlap.

18. The method of claim 17 further comprising;

generating a fourth source of binocular video feed of a traveling matte by extracting portions of the first binocular video feed; and

subtracting the fourth source of traveling matte binocular video feed from a fifth source of binocular video feed of video game imagery written on a model of the physical environment to form the third source of binocular video feed.

19. The method of claim 18 further comprising:

tracking the position of the headset with at least one camera mounted to the headset by imaging indicia written on the physical environment and visible to the at least one tracking camera;

creating a CAD model of the physical environment and aligning the CAD model with a tracked position of the headset;

producing a sixth source of video feed of the CAD model from a view of the model corresponding to the first source of binocular video feed of the physical environment;

creating video game imagery and writing the video game imagery to selected portions of the CAD model of the physical environment and creating the fifth source of binocular video feed corresponding to the view of the model corresponding to the first source of binocular video feed of the physical environment and having the video game imagery written thereon; and

wherein the fourth source of traveling matte binocular video is created by comparing the first source of binocular video feed of the physical environment to the sixth source of video feed view of the CAD model to generate the traveling matte from portions of the physical environment which lack correspondence to the CAD model.

20. The method of claim 19 wherein the fourth source of traveling matte binocular video is created by comparing the first source of binocular video feed to the sixth source of binocular video feed CAD model by machine vision and drawing an outline of portions of the physical environment which lack correspondence to the CAD model and filling the outline.

21. The method of claim 19 further comprising the step of firing a laser beam from the gun carried by the trainee and locating a hit point for the laser beam in the physical environment by identifying light of the laser beam in the first source of binocular video feed.

22. The method of claim 19 further comprising the step of generating a training scenario, and using the training scenario to determine the videogame imagery, and the selected portions of the CAD model on which the videogame imagery appears in the second video feed as displayed on the binocular display mounted to the headset.

23. The method of claim 22 wherein the generated training scenario includes a time or event line over which the videogame imagery changes.

24. The method of claim 19 further comprising the step of matching brightness of the first source of binocular video feed with the brightness of the third source of binocular video feed.