Method for displaying emergency first responder command, control, and safety information using augmented reality

• US 20020196202A1
• Filed: 08/09/2002
• Published: 12/26/2002
• Est. Priority Date: 08/09/2000
• Status: Abandoned Application

• Current Assignee: Information Decision Technologies LLC
• Original Assignee: Information Decision Technologies LLC
• Inventors: Eads, Daniel Alan, Ebersole, John Franklin, Ebersole, John Franklin Jr., Bastian, Mark Stanley
• Application Number:
  US10/216,304
• Publication Number:
  US 20020196202A1
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A method is presented which uses Augmented Reality for visualization of text and other messages sent to an EFR by an incident commander. The messages are transmitted by the incident commander via a computer at the scene to an EFR/trainee in an operational or training scenario. Messages to an EFR/trainee, including (but not limited to) iconic representation of hazards, victims, structural data, environmental conditions, and exit directions/locations, are superimposed right onto an EFR/trainee'"'"'s view of the real emergency/fire and structural surroundings. The primary intended applications are for improved safety for the EFR, and improved EFR-incident commander communications both on-scene and in training scenarios.

Claims

1. A method for displaying emergency first responder command, control, and safety information comprising:

providing a display device;

obtaining data about the current physical location of the display device;

obtaining data about the current orientation of the display device;

generating 2D and 3D information for the user of the display device by using a computer;

transmitting the information to a computer worn or held by the user;

rendering 3D graphical elements based on the 3D information on the computer worn or held by the user;

creating an overlay of the 2D information on the computer worn or held by the user;

creating for the user a mixed view comprised of an actual view of the real world as it appears in front of the user, where 3D graphical elements can be placed any place in the real world that can be anchored to that place in the real world regardless of the direction in which the user is looking, wherein the rendered 3D graphical elements and 2D information are superimposed on the actual view, to accomplish an augmented reality view.

2. The method of claim 1 in which the user display device is selected from the group of display devices consisting of a Head Mounted Display (HMD), a see-through HMD, a non-see-through HMD, a monocular type of HMD, an HMD integrated into the user'"'"'s face mask, a hand held display device, a see-through device, and a non-see through device.

3. The method of claim 2 in which the real world image is obtained using a video camera.

4. The method of claim 2 in which the face mask is selected from the group of face masks consisting of a firefighter'"'"'s SCBA (Self Contained Breathing Apparatus), a face mask that is part of a HAZMAT (Hazardous Materials) suit, a face mask that is part of a radiation suit, and a face mask that is part of a hard hat.

5. The method of claim 2 in which the non-see-through display device obtains an image of the real world using a video camera.

6. The method of claim 2 in which the hand held device is integrated into another device.

7. The method of claim 6 in which the other device is selected from the group of devices consisting of a Thermal Imager, a Navy Firefighter'"'"'s Thermal Imager (NFTI), and a Geiger counter.

8. The method of claim 1 in which the information transmitted to the user'"'"'s computer is selected from the group of information consisting of textual data, directional navigation data, iconic information, and a wireframe view of the incident space in which the user is physically located.

9. The method of claim 1 in which the rendered data is selected from the group of rendered data consisting of navigation data, telling the user the direction in which to travel, warning data, telling the user of dangers of which the user may not be aware, environmental temperature at the location of the user, environmental temperature at a location the user is approaching, information pertaining to the area in which the event is occurring to help the user safely and thoroughly perform a task, information pertaining to individuals at an incident site, and an arrow that the user can follow to reach a destination.

10. The method of claim 1 in which a waypoint mode is established in which direction-indicating icons are displayed on the computer worn or held by the user, to create for the user intermediate points along a path that the user can follow in order to reach a final destination.

11. The method of claim 10 in which an icon is displayed to indicate the final destination of the user along the waypoint path.

12. The method of claim 10 in which icons are displayed to represent intermediate points between the user'"'"'s current location and final destination.

13. The method of claim 10 in which the icon about information is used to represent harmful hazards that are located in an area, the harmful hazard selected from the group of hazards consisting of a fire, a bomb, a radiation leak, and a chemical spill.

14. The method of claim 1 in which the information transmitted to the user'"'"'s wearable computer originates from a user operating a computing device.

15. The method of claim 8 in which a model is used to show the wireframe representation, wherein the model is obtained from a geometric model created before the time of use.

16. The method of claim 8 in which a model is used to show the wireframe representation, wherein the model is generated at the time of use.

17. The method of claim 16 in which equipment mounted on the user is used to generate the wireframe model of the space.

18. The method of claim 17 in which the model is generated as the user traverses the space.

19. The method of claim 16 in which the equipment used to generate the model of the space the user is in is carried by the user.

20. The method of claim 16 in which the equipment used to generate the model of the space the user is in is on a stationary mount.

21. The method of claim 16 in which the model obtained at the time of use is shared with other users.

22. The method of claim 21 in which the model of the space is shared with other users using wireless connections.

23. The method of claim 21 in which the model of the space is shared with other users using wired connections.

24. The method of claim 21 in which the shared model information is used in conjunction with other model information to create an enlarged model.

25. The method of claim 24 in which the enlarged model is shared and can be used by other users.

26. The method of claim 1 in which obtaining data about the current location and orientation of the display device comprises using a radio frequency tracking technology.

27. The method of claim 26 in which there are at least three radio frequency transmitters located in proximity to the space the user is in, and where the user has a radio frequency receiver.

28. The method of claim 27 in which the radio frequency receiver determines the direction of each of the radio frequency transmitters, and from that it determines the location of the user relative to the transmitter.

29. The method of claim 26 in which the radio frequency receiver determines the distance to each of the radio frequency transmitters, and from that information determines the location of the user relative to the transmitter.

30. The method of claim 26 in which there are at least three radio frequency receivers located in proximity to the space the user is in, and where the user has a radio frequency transmitter on his/her person.

31. The method of claim 30 in which the radio frequency receivers determine the direction of the radio frequency transmitter, and from that determine the location of the user relative to the receivers.

32. The method of claim 30 in which the radio frequency receivers determine the distance of the radio frequency transmitter, and from that information determine the location of the user relative to the receivers.

33. The method of claim 26 in which the tracking equipment on the user is selected from the group of tracking equipment consisting of a compass-type unit that determines the direction of magnetic north, which is used to determine the orientation of the display device relative to the stationary receivers/transmitters, tracking equipment on the user that has two receiver/transmitter units, which are used to determine the orientation of the display device relative to the stationary receivers/transmitters, and tracking equipment on the user that has a tilt sensor that senses tilt in two axes, thereby allowing the tracking technology to know roll and pitch of the user.

34. The method of claim 1 in which the positions of at least one user is shared with others.

35. The method of claim 34 in which the user can see a display of the positions of other users in the space.

36. The method of claim 34 in which the positions of a user are recorded.

37. The method of claim 36 in which the user can see a display of his/her path taken through the space.

38. The method of claim 36 in which a user can see a display of the paths of other users taken through the space.

39. The method in claim 1 in which the method is used in operations.

40. The method in claim 1 in which the method is used in training.

41. The method in claim 1 in which the user is selected from the group of users consisting of an emergency first responder, an outside observer, and an incident commander.