Augmented reality maintenance system with flight planner

• US 5,550,758 A
• Filed: 03/29/1994
• Issued: 08/27/1996
• Est. Priority Date: 03/29/1994
• Status: Expired due to Term

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Solanas, Claude H. III, Meenan, Peter M., Vickerman, David C., Corby, Nelson R. Jr.
• Primary Examiner(s): Trammell, James P.
• Application Number:
  US08/219,558
• Time in Patent Office: 882 Days
• Field of Search: 364/571.03, 364/424.01, 364/424.02, 364/402, 364/433, 364/516, 364/525, 364/551.01, 364/559, 364/570, 364/571.4, 364/578, 395/94, 395/97, 395/907, 395/930, 367/6, 367/48, 367/907, 367/134, 358/103, 358/108
• US Class Current: 702/150
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005

Abstract

An enhanced reality maintenance system for operating in a hazardous or inaccessible environment employs an environment modeler which is supplied with spatial parameters of a given environment and creates a computer model of the environment. An environment renderer creates a plurality of images, each corresponding to a viewing location and orientation, `viewpoint`. A remotely operated vehicle (ROV) attached to a base unit by a tether cord or radio link navigates in the environment. The ROV has a spatial imaging device, such as a video camera, and actuators which propel it through the environment. Its position and orientation are determined by a position and attitude (P&A) sensing unit, and are passed to an ROV renderer which creates an image of a prestored model of the ROV having the same location and orientation as the ROV and viewed from a specified viewpoint. The viewpoints may be predetermined, provided to the system or may be interactively determined as an offset from the ROV position and orientation. Alternative embodiments include an image archive and comparison unit capable of storing images linked to information of the image acquisition, retrieving stored images with the image acquisition into and transforming one of the images to match the image acquisition information of the other image.

Claims

1. A remote maintenance system for inspection and repair of structures in an environment comprising:

a) a remotely operated vehicle (ROV) having

1. a sensor package, capable of inspecting physical characteristics of said structures in environment, and

2. actuators capable of maneuvering the ROV;

b) a position and attitude (P&

A) sensing unit for providing a position and orientation of the ROV;

c) environmental model storage device capable of retaining computer graphic information regarding the environment of the ROV;

d) environment modeler for receiving parameters defining objects in an environment of the ROV and producing a computer generated model of the environment and storing the model in the environment model storage device;

e) environment renderer coupled to the environment model storage device for generating an image of the environment from the environment model as viewed from at least one viewpoint;

f) ROV renderer coupled to the P&

A sensing unit adapted for generating an image of the ROV at a position and orientation corresponding to that of the ROV, generated from a prestored model of the ROV as viewed from at least one viewpoint;

g) monitor for displaying a video signal;

h) video mixer coupled to the environment renderer and the ROV renderer for displaying the images on the monitor, thereby allowing an operator to visualize the position of the ROV relative to its environment; and

i) an automated flight planner comprised of;

i. a pointing device allowing the operator to select points on a path which the operator desires the ROV to follow;

ii. a flight path storage unit capable of storing information defining a path through space; and

iii. a trajectory computer coupled to the environment model storage device, the pointing device, the flight path storage unit, functionally coupled to the ROV renderer and the actuators, for approximating a path closest to the points selected by the operator which does not cause the ROV to collide with objects in the environment, being a selected trajectory, storing this trajectory in the flight path storage unit, and causing the ROV to follow the selected trajectory.

2. The remote maintenance system of claim 1 further comprising viewpoint unit coupled to the ROV renderer and the environment renderer which provides the viewpoint employed in image rendering.

3. The remote maintenance system of claim 1 further comprising offset computation device coupled to the P&

A sensing unit which provides the viewpoint, being an offset to location and orientation of the ROV received from the P&

A sensing unit, employed by the environment renderer and the ROV renderer in image rendering.

4. The remote maintenance system of claim 1 further comprising a path execution unit coupled to the P&

A unit and the flight path storage unit and actuators for reading the selected trajectory from the flight path storage unit and causing the actuators to move the ROV along that path.

5. The remote maintenance system of claim 1 further comprising a flight path monitoring unit coupled to the flight path storage unit, the P&

A unit and the video mixer, for reading the selected trajectory from the flight path storage unit, and monitoring the actual location and position of the ROV over time, indicating the ROV'"'"'s actual path, and displaying the selected trajectory and the actual path of the ROV on the monitor.