Augmented reality-based firefighter training system and method
First Claim
Patent Images
1. A method of accomplishing an augmented reality firefighter training system for a user, comprising:
- providing a head-worn display unit;
providing a real device that the user can operate to simulate applying extinguishing agent, to be carried by the user during firefighter training;
providing motion tracking hardware, and attaching it both to the head-worn display unit and the extinguishing agent device;
using the motion tracking hardware that is attached to the head worn unit to determine the location and direction of the viewpoint of the head-worn display unit;
using the motion tracking hardware that is attached to the extinguishing agent device to determine the location and direction of the extinguishing agent device;
determining the operating state of the extinguishing agent device;
using a computer to generate graphical elements comprising simulated fire graphical elements, simulated multiple layer smoke obscuration graphical elements, and simulated application of an extinguishing agent, showing the extinguishing agent itself emanating directly from the extinguishing agent device, and showing the interaction and extinguishing of the agent with the fire;
rendering the generated graphical elements to correspond to the user'"'"'s viewpoint; and
creating for the user a mixed view comprised of an actual view of the real world as it appears in front of the user, where graphical elements can be placed any place in the real world and remain anchored to that place in the real world regardless of the direction in which the user is looking, wherein the rendered graphical elements are superimposed on the actual view, to accomplish an augmented reality view of fire and smoke in the real world, and the application of extinguishing agent to the fire, and the effect of extinguishing agent on the fire.
3 Assignments
0 Petitions
Accused Products
Abstract
Method and apparatus are presented for an augmented reality-based firefighter training system. The system includes hardware for motion tracking, display, and vari-nozzle instrumentation. System software includes a real-time fire model, a layered smoke obscuration model, simulation of an extinguishing agent, and an interface to a zone fire model. Physical modeling and graphical elements in the software combine to create realistic-looking fire, smoke, and extinguishing graphics. The hardware and software components together contribute to a realistic, interactive training experience for firefighters.
-
Citations
19 Claims
-
1. A method of accomplishing an augmented reality firefighter training system for a user, comprising:
-
providing a head-worn display unit;
providing a real device that the user can operate to simulate applying extinguishing agent, to be carried by the user during firefighter training;
providing motion tracking hardware, and attaching it both to the head-worn display unit and the extinguishing agent device;
using the motion tracking hardware that is attached to the head worn unit to determine the location and direction of the viewpoint of the head-worn display unit;
using the motion tracking hardware that is attached to the extinguishing agent device to determine the location and direction of the extinguishing agent device;
determining the operating state of the extinguishing agent device;
using a computer to generate graphical elements comprising simulated fire graphical elements, simulated multiple layer smoke obscuration graphical elements, and simulated application of an extinguishing agent, showing the extinguishing agent itself emanating directly from the extinguishing agent device, and showing the interaction and extinguishing of the agent with the fire;
rendering the generated graphical elements to correspond to the user'"'"'s viewpoint; and
creating for the user a mixed view comprised of an actual view of the real world as it appears in front of the user, where graphical elements can be placed any place in the real world and remain anchored to that place in the real world regardless of the direction in which the user is looking, wherein the rendered graphical elements are superimposed on the actual view, to accomplish an augmented reality view of fire and smoke in the real world, and the application of extinguishing agent to the fire, and the effect of extinguishing agent on the fire. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
employing a first particle system for lower persistent flames;
employing a second particle system for upper intermittent flames;
texture mapping both of said particle systems;
giving flame particles an impetus towards the flame center to simulate air entrainment; and
employing a third particle system for a smoke plume.
-
-
4. The method of claim 1 in which the smoke obscuration graphical elements rendering is accomplished by:
-
displaying polygonal representations of objects;
calculating transmittance along a vector from the viewpoint to the polygons;
applying said transmittance, original polygon color, and obscuration color to recolor the polygons; and
modifying said transmittance to represent light sources.
-
-
5. The method of claim 1 in which the extinguishing agent is simulated using the following method:
-
employing a particle system to simulate physical behavior of an extinguishing agent stream;
organizing particles into regularly spaced rings;
using particle locations as polygon vertices to create a surface;
texture mapping said surface; and
moving said texture map in the direction of extinguishing agent flow to simulate extinguishing agent flow.
-
-
6. The method of claim 1 in which real-world imagery is provided by a head-worn camera.
-
7. The method of claim 6 in which real-world imagery and the rendered graphical elements are mixed via a luminance key in a video mixer.
-
8. The method of claim 7 in which a separate image indicating the transparency of the rendered graphical elements is generated and used as an external key.
-
9. The method of claim 6 in which real-world imagery and the rendered graphical elements are mixed on a computer using a transparency (alpha) channel for blending.
-
10. The method of claim 1 in which real-world imagery and the rendered graphical elements are combined on an optical see-through head-worn display.
-
11. The method of claim 1 in which multiple trainees with similar apparatus can view and interact with the same fire scenario.
-
12. The method of claim 1 further comprising creating a geometric representation of real-world objects and using the geometric representation to occlude the computer-generated graphical elements so as to allow them to appear appropriately behind or in front of real-world objects.
-
13. The method of claim 1 in which the rendering comprises matching the user'"'"'s virtual eye location and field of view to the user'"'"'s real eye location and field of view.
-
14. The method of claim 1 in which view frustum culling is applied to the computer-generated graphical elements to optimize graphics performance.
-
15. The method of claim 1 in which the device that the user can operate to simulate applying extinguishing agent is instrumented to allow adjustment of the flow rate and fog pattern of the computer-generated extinguishing agent.
-
16. The method of claim 2 in which the upper layer temperature is monitored and an indication of flashover is displayed when it is predicted to occur.
-
17. The method of claim 1 further comprising attaching audio to the location of objects in the mixed view.
-
18. The method of claim 17 in which the volume of the audio increases with fire size.
-
19. The method of claim 1 further comprising periodically updating the prediction amount for motion tracking based on display frame rate in order to best anchor graphical elements in the real environment.
Specification