Augmented reality-based firefighter training system and method

• US 6,500,008 B1
• Filed: 03/15/2000
• Issued: 12/31/2002
• Est. Priority Date: 03/15/1999
• Status: Expired due to Fees

• Current Assignee: Information Decision Technologies LLC
• Original Assignee: Information Decision Technologies LLC
• Inventors: Ebersole, John Franklin, Ebersole, John Franklin Jr., Furlong, Todd Joseph
• Primary Examiner(s): Cheng, Joe H.
• Assistant Examiner(s): MOSSER, KATHLEEN MICHELE
• Application Number:
  US09/525,983
• Time in Patent Office: 1,021 Days
• Field of Search: 434/226, 434/11, 434/29-38, 434/234, 434/307.R, 345/426, 345/441, 345/112
• US Class Current: 434/226
• CPC Class Codes: A62C 99/0081 Training methods or equipment for fire-fighting
• CPC Class Codes: G06T 13/60 of natural phenomena, e.g. rain, snow, water or plants
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26

Abstract

Method and apparatus are presented for an augmented reality-based firefighter training system. The system includes hardware for motion tracking, display, and vari-nozzle instrumentation. System software includes a real-time fire model, a layered smoke obscuration model, simulation of an extinguishing agent, and an interface to a zone fire model. Physical modeling and graphical elements in the software combine to create realistic-looking fire, smoke, and extinguishing graphics. The hardware and software components together contribute to a realistic, interactive training experience for firefighters.

Claims

1. A method of accomplishing an augmented reality firefighter training system for a user, comprising:

providing a head-worn display unit;

providing a real device that the user can operate to simulate applying extinguishing agent, to be carried by the user during firefighter training;

providing motion tracking hardware, and attaching it both to the head-worn display unit and the extinguishing agent device;

using the motion tracking hardware that is attached to the head worn unit to determine the location and direction of the viewpoint of the head-worn display unit;

using the motion tracking hardware that is attached to the extinguishing agent device to determine the location and direction of the extinguishing agent device;

determining the operating state of the extinguishing agent device;

using a computer to generate graphical elements comprising simulated fire graphical elements, simulated multiple layer smoke obscuration graphical elements, and simulated application of an extinguishing agent, showing the extinguishing agent itself emanating directly from the extinguishing agent device, and showing the interaction and extinguishing of the agent with the fire;

rendering the generated graphical elements to correspond to the user'"'"'s viewpoint; and

creating for the user a mixed view comprised of an actual view of the real world as it appears in front of the user, where graphical elements can be placed any place in the real world and remain anchored to that place in the real world regardless of the direction in which the user is looking, wherein the rendered graphical elements are superimposed on the actual view, to accomplish an augmented reality view of fire and smoke in the real world, and the application of extinguishing agent to the fire, and the effect of extinguishing agent on the fire.

2. The method of claim 1 in which flame height and smoke layer parameters of the graphical elements are controlled by output from a zone fire model.

3. The method of claim 1 in which the simulated fire graphical elements are generated by:

employing a first particle system for lower persistent flames;

employing a second particle system for upper intermittent flames;

texture mapping both of said particle systems;

giving flame particles an impetus towards the flame center to simulate air entrainment; and

employing a third particle system for a smoke plume.

4. The method of claim 1 in which the smoke obscuration graphical elements rendering is accomplished by:

displaying polygonal representations of objects;

calculating transmittance along a vector from the viewpoint to the polygons;

applying said transmittance, original polygon color, and obscuration color to recolor the polygons; and

modifying said transmittance to represent light sources.

5. The method of claim 1 in which the extinguishing agent is simulated using the following method:

employing a particle system to simulate physical behavior of an extinguishing agent stream;

organizing particles into regularly spaced rings;

using particle locations as polygon vertices to create a surface;

texture mapping said surface; and

moving said texture map in the direction of extinguishing agent flow to simulate extinguishing agent flow.

6. The method of claim 1 in which real-world imagery is provided by a head-worn camera.

7. The method of claim 6 in which real-world imagery and the rendered graphical elements are mixed via a luminance key in a video mixer.

8. The method of claim 7 in which a separate image indicating the transparency of the rendered graphical elements is generated and used as an external key.

9. The method of claim 6 in which real-world imagery and the rendered graphical elements are mixed on a computer using a transparency (alpha) channel for blending.

10. The method of claim 1 in which real-world imagery and the rendered graphical elements are combined on an optical see-through head-worn display.

11. The method of claim 1 in which multiple trainees with similar apparatus can view and interact with the same fire scenario.

12. The method of claim 1 further comprising creating a geometric representation of real-world objects and using the geometric representation to occlude the computer-generated graphical elements so as to allow them to appear appropriately behind or in front of real-world objects.

13. The method of claim 1 in which the rendering comprises matching the user'"'"'s virtual eye location and field of view to the user'"'"'s real eye location and field of view.

14. The method of claim 1 in which view frustum culling is applied to the computer-generated graphical elements to optimize graphics performance.

15. The method of claim 1 in which the device that the user can operate to simulate applying extinguishing agent is instrumented to allow adjustment of the flow rate and fog pattern of the computer-generated extinguishing agent.

16. The method of claim 2 in which the upper layer temperature is monitored and an indication of flashover is displayed when it is predicted to occur.

17. The method of claim 1 further comprising attaching audio to the location of objects in the mixed view.

18. The method of claim 17 in which the volume of the audio increases with fire size.

19. The method of claim 1 further comprising periodically updating the prediction amount for motion tracking based on display frame rate in order to best anchor graphical elements in the real environment.