Method for automatically tracking objects in augmented reality

• US 7,138,963 B2
• Filed: 04/16/2004
• Issued: 11/21/2006
• Est. Priority Date: 07/18/2002
• Status: Expired due to Fees

• Current Assignee: MetaMersion, LLC
• Original Assignee: MetaMersion, LLC
• Inventors: Hobgood, Andrew W., Ebersole, John Franklin, Ebersole, John Franklin Jr.
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): DINH, DUC Q
• Application Number:
  US10/825,763
• Publication Number:
  US 20040212630A1
• Time in Patent Office: 949 Days
• Field of Search: 345/115, 345 7- 9
• US Class Current: 345/8
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

The invention is a method for displaying otherwise unseen objects and other data using augmented reality (the mixing of real view with computer generated imagery). The method uses a motorized camera mount that can report the position of a camera on that mount back to a computer. With knowledge of where the camera is looking, and the size of its field of view, the computer can precisely overlay computer-generated imagery onto the video image produced by the camera. The method may be used to present to a user such items as existing weather conditions, hazards, or other data, and presents this information to the user by combining the computer generated images with the user'"'"'s real environment. These images are presented in such a way as to display relevant location and properties of the object to the system user. The primary intended applications are as navigation aids for air traffic controllers and pilots in training and operations, and use with emergency first responder training and operations to view and avoid/alleviate hazardous material situations, however the system can be used to display any imagery that needs to correspond to locations in the real world.

Claims

1. A method for automatically tracking an object to keep the object in the field of view of a video camera, and create an augmented reality display comprising the image from the video camera combined with computer-generated graphics to create an augmented reality display, the method comprising:

providing a motorized camera mount for the camera, and controlling the camera mount using a computer;

resolving the camera'"'"'s field of view;

determining the location of an object to be tracked relative to the camera'"'"'s field of view;

in response to the determined location of the object, received by the computer from the motorized camera mount, moving the camera to maintain the object in the camera'"'"'s field of view;

wherein the resolving step comprises using the computer to resolve the field of view based on the current camera position received by the computer from the camera mount, to accomplish a feedback control system;

using the computer to generate a graphical image representing unseen information that corresponds to the camera'"'"'s viewpoint; and

augmenting the video image with the graphical image, to create an augmented-reality image having the tracked object therein, for presentation to a user.

2. The method of claim 1 wherein the augmenting step comprises using onboard video mixing through use of a video capture device with the computer.

3. The method of claim 1 wherein the augmenting step comprises using an external video mixing solution, to combine real and computer-generated graphical elements outside of the computer.

4. The method of claim 1 for use in operations.

5. The method of claim 1 for use in training.

6. The method of claim 1 in which the resolving step comprises calibrating the camera and camera mount.

7. The method of claim 1 in which the camera mount is coupled to a fixed platform.

8. The method of claim 1 in which the resolving step comprises using the camera and camera mount in conjunction with a separate camera position tracking system to generate a combined position and orientation value.

9. The method of claim 1 in which the computer controls the camera mount, and the resolving step comprises using the computer to position the camera in a feed-forward control system.

10. The method of claim 1 in which the camera mount is not stationary.

11. The method of claim 10 in which the camera mount is attached to a vehicle.

12. The method of claim 10 in which the camera mount is attached to an aircraft.

13. The method of claim 10 in which the camera mount is attached to a watercraft or ship.

14. The method of claim 10 in which the camera mount is attached to a gimbaled arm.

15. The method of claim 1 in which the resolving step comprises the motorized camera mount reporting the field of view of the camera to the computer.

16. A method for automatically tracking an object to keep the object in the field of view of a video camera, and create an augmented reality display comprising the image from the video camera combined with computer-generated graphics to create an augmented reality display, the method comprising:

providing a motorized camera mount for the camera;

resolving the camera'"'"'s field of view;

determining the location of an object to be tracked relative to the camera'"'"'s field of view;

in response to the determined location of the object, moving the camera to maintain the object in the camera'"'"'s field of view;

using a computer to control the camera mount, and wherein the resolving step comprises using the computer to resolve the field of view based on the current camera position received by the computer from the camera mount, to accomplish a feedback control system;

using the computer to generate a graphical image representing unseen information that corresponds to the camera'"'"'s viewpoint;

using the computer to control the camera'"'"'s viewpoint; and

using a video capture device along with the computer to accomplish video mixing that augments the video image with the graphical image, to create an augmented-reality image having the tracked object therein, for presentation to a user.