Method and system for accomplishing a scalable, multi-user, extended range, distributed, augmented reality environment

• US 7,046,214 B2
• Filed: 03/29/2005
• Issued: 05/16/2006
• Est. Priority Date: 12/17/2003
• Status: Expired due to Fees

• Current Assignee: Information Decision Technologies LLC
• Original Assignee: Information Decision Technologies LLC
• Inventors: Ebersole, John Franklin, Ebersole, John Franklin Jr., Hobgood, Andrew Wesley
• Primary Examiner(s): Shankar, Vijay
• Assistant Examiner(s): Patel, Nitin
• Application Number:
  US11/092,083
• Publication Number:
  US 20050168403A1
• Time in Patent Office: 413 Days
• Field of Search: 345/7, 345/8, 345/9, 345/145, 345/146, 345/157, 345340-341, 345/348, 345/421, 345/419, 345/473, 434/226, 434/307
• US Class Current: 345/7
• CPC Class Codes: A62B 18/04 Gas helmets
• CPC Class Codes: A62B 9/006 Indicators or warning devices, e.g. of low pressure, contamination
• CPC Class Codes: A62C 99/0081 Training methods or equipment for fire-fighting
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/017 Head mounted

Abstract

A wireless or extra-long-tethered augmented reality (AR) system and method, where the user wears some or all of the equipment necessary to perform the simulation. Various arrangements are presented that can be selected based on the needs of the system, such as the number of users and type of tracking equipment. Most of the discussion is optimized to a firefighter training system, but the disclosure is applicable to most any other application of wireless AR.

Claims

1. A method for accomplishing a scalable, multi-user, extended range, distributed, augmented reality environment, comprising:

providing a head-worn display unit;

providing a computer system and video hardware worn by the user;

providing a data communications system worn by the user to permit the computer to communicate with other computer systems;

providing motion tracking hardware worn by the user;

using the motion tracking hardware to determine the position and orientation of the user'"'"'s viewpoint;

using the data communications system to send or receive current user position, simulation modification command made by users or instructors, and computer-driven simulation status to other systems participating in the environment;

using the computer system to simulate a synthetic environment comprising graphical elements;

using the computer system and video hardware to render the graphical elements corresponding to the user'"'"'s viewpoint; and

creating for the user a mixed view comprised of an actual view of the real world as it appears from the user'"'"'s viewpoint, where graphical elements can be placed any place in the real world and remain anchored to that place in the real world regardless of the direction in which the user is looking, wherein the rendered graphical elements are superimposed on the actual view, to accomplish an augmented reality view of the simulated environment.

2. The method of claim 1 in which the environment is a firefighter training environment, and a simulated fire may be extinguished by the user, comprising:

specifying that the environment be a firefighter training environment;

providing a physical device that the user can operate to simulate applying extinguishing agent, the device carried by the user during firefighter training;

providing motion tracking hardware attached to the extinguishing agent application device;

using the motion tracking hardware that is attached to the extinguishing agent application device to determine the position and orientation of the extinguishing agent application device;

determining the operating state of the extinguishing agent application device;

transmitting the position, orientation, and operating state of the extinguishing agent application device to other systems participating in the firefighting environment; and

using the position, orientation, and operating state of the extinguishing agent application device to generate graphical elements comprising simulated application of an extinguishing agent, showing the extinguishing agent itself emanating directly from the extinguishing agent application device, and showing the interaction and extinguishing of fire with the agent.

3. The method of claim 2 where the extinguishing agent application device is selected from the group consisting of a water nozzle, AFFF nozzle, PKP extinguisher, CO2 extinguisher, and AFFF extinguisher.

4. The method of claim 2 where the extinguishing agent application device is operated wirelessly.

5. The method of claim 1 in which the environment is a police or military training environment, comprising:

specifying that the environment be a military-style training environment;

providing a physical device that the user can operate that simulates the functionality of a gun, which is carried by the user during training;

providing motion tracking hardware attached to the physical device;

using the motion tracking hardware that is attached to the physical device to determine the position and orientation of the device;

determining the operating state of the device;

transmitting the position, orientation, and operating state of the device to other systems participating in the environment; and

using the position, orientation, and operating state of the device to generate graphical elements comprising simulated gunfire, and showing the results of the gunfire on elements in the simulation.

6. The method of claim 1 in which a tether is used to connect at least some of the user-worn equipment to a remotely-mounted set of equipment to provide at least one of power and data to the user-worn equipment.

7. The method of claim 1 in which at least one battery is carried by the user, to provide power for at least some of the other equipment carried by the user.

8. The method of claim 1 in which the power used by the various user-worn items is generated on the user, from a single power source provided over a tether connected between the user-worn items and a remote location.

9. The method of claim 1 in which at least one power supply creates at least one voltage output.

10. The method of claim 1 in which the unit is operated wirelessly, and has no tether.

11. The method of claim 10 in which the equipment worn by the user transmits tracking data and video data to a remote location, and the necessary computer-generated imagery is created and mixed with the video data at the remote location, and then the mixed augmented reality video is transmitted back to the equipment worn by the user for display to the user.

12. The method of claim 10 in which the equipment worn by the user transmits tracking data to a remote location, and the necessary computer-generated imagery is produced at the remote location and then transmitted back to the equipment worn by the user, and the equipment worn by the user mixes the computer-generated imagery with the video created by the video hardware worn by the user, thus generating the finished augmented reality video for display to the user.

13. The method of claim 10 in which the equipment worn by the user transmits tracking data to a remote location, and tracking position data is computed at the remote location and then transmitted to the equipment worn by the user, and the equipment worn by the user produces the necessary computer-generated imagery, and mixes the computer-generated imagery with the video created by the video hardware worn by the user, thus generating the finished augmented reality video for display to the user.

14. The method of claim 13 in which the video created by the video hardware worn by the user is captured by the computer worn by the user, and then the computer mixes the computer-generated imagery with the video.

15. The method of claim 10 in which the equipment worn by the user performs all necessary processing of video data and tracking data, without the assistance of any remote computer hardware.

16. The method of claim 1 in which software-based video mixing is performed, where the computer captures a video stream, renders computer-generated simulation components, mixes them with the video stream, and then outputs the completed augmented reality video stream.

17. A method of tracking a user, in which equipment worn by a user performs user-tracking processing, comprising:

providing active motion tracking hardware installed in the environment;

providing motion tracking hardware worn by the user which is able to receive signals from the active motion tracking hardware installed in the environment, and those signals contain information about the relative position of the motion tracking hardware worn by the user in relation to the active motion tracking hardware installed in the environment;

providing motion tracking data processing hardware worn by the user which processes the signals received by the motion tracking hardware worn by the user;

providing a means for the motion tracking data processing hardware to receive information about the signals of the active motion tracking hardware installed in the environment, where the information at least indicates when the motion tracking hardware installed in the environment is emitting a signal; and

using the motion tracking hardware and additional motion tracking data processing hardware to determine the position and orientation of the user'"'"'s viewpoint in relation to the motion tracking hardware installed in the environment.

18. The method of claim 17 in which the means to receive said information about the signals of the active motion tracking hardware installed in the environment is accomplished with a wireless link.