Augmented reality-based system and method providing status and control of unmanned vehicles

• US 7,920,071 B2
• Filed: 03/08/2007
• Issued: 04/05/2011
• Est. Priority Date: 05/26/2006
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: ITT Manufacturing Enterprises, Inc. (ITT, Inc.)
• Inventors: Baillot, Yohann
• Primary Examiner(s): POPE, DARYL C
• Application Number:
  US11/715,339
• Publication Number:
  US 20070273557A1
• Time in Patent Office: 1,489 Days
• Field of Search: 340/573.1, 340/10.34, 340/539.13, 342/44, 342/40, 345/7, 345/633, 345/632, 345/8, 375/239, 701/207, 702/186
• US Class Current: 340/937
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

An augmented reality system identifies and controls a vehicle located within an environment. A tracking system obtains viewpoint information corresponding to a real-time view of the environment. A processing system receives information from one or more sensors. Information includes sensor location information and status information about the vehicle. Processing system generates graphics using said sensor location information and said viewpoint information. Graphics include visual representations of said status information and controls. A display displays the generated graphics such that the graphics are superimposed on the real-time view. The graphics appear attached to the vehicle. An interaction device activates a displayed control.

Claims

1. A method of identifying and controlling a vehicle located within an environment comprising the steps of:

receiving information from one or more sensors coupled to the vehicle, where the information includes sensor location information and status information about the vehicle;

obtaining viewpoint information corresponding to a real-time view of said environment;

generating graphics using said sensor location information and viewpoint information, wherein the graphics include visual representations of the status information and controls;

displaying the generated graphics on a display such that the graphics are superimposed on the real-time view, wherein the graphics appear attached to the vehicle; and

activating a displayed control.

2. The method of claim 1, wherein visual representations of status information include:

an identifier that identifies the vehicle.

3. The method of claim 1, wherein the displayed control includes:

an action button that when activated displays information captured from a number of said one or more sensors coupled to the vehicle.

4. The method of claim 1, wherein the displayed control includes:

arrows that upon being selected move the vehicle in the direction selected.

5. The method of claim 1, wherein the displayed control includes:

an icon next to a vehicle that upon being moved to a location moves the vehicle to a corresponding location.

6. The method of claim 1, wherein the displayed control includes:

an action button next to a vehicle that upon being activated commands the vehicle to move to a predetermined location.

7. The method of claim 1, wherein the displayed control includes:

an action button next to a vehicle that upon being activated commands the vehicle to perform a series of operations.

8. An augmented reality system for identifying and controlling a vehicle located within an environment comprising:

one or more sensors coupled to the vehicle;

a tracking system that obtains viewpoint information corresponding to a real-time view of said environment;

a processing system that receives information from said one or more sensors, where the information includes sensor location information and status information about the vehicle, and generates graphics using said sensor location information and said viewpoint information, wherein the graphics include visual representations of said status information and controls;

a display that displays the generated graphics such that the graphics are superimposed on the real-time view, wherein the graphics appear attached to the vehicle; and

an interaction device that activates a displayed control.

9. The augmented reality system of claim 8, wherein visual representations of status information include:

an identifier that identifies the vehicle.

11. The augmented reality system of claim 8, wherein the displayed control includes:

arrows that upon being selected move the vehicle in the direction selected.

12. The augmented reality system of claim 8, wherein the displayed control includes:

an icon next to a vehicle that upon being moved to a location moves the vehicle to a corresponding location.

13. The augmented reality system of claim 8, wherein the displayed control includes:

an action button next to a vehicle that upon being activated commands the vehicle to move to a predetermined location.

14. The augmented reality system of claim 8, wherein the displayed control includes:

an action button next to a vehicle that upon being activated commands the vehicle to perform a series of operations.

10. The augmented reality system of 8, wherein the displayed control includes:

an action button that when activated displays information captured from a number of said one or more sensors coupled to the vehicle.

15. An augmented reality system for identifying and controlling an apparatus located within an environment comprising:

one or more sensors coupled to the apparatus;

a tracking system that obtains viewpoint information corresponding to a real-time view of said environment;

a processing system that receives information from said one or more sensors, where the information includes sensor location information and status information about the apparatus, and generates graphics using said sensor location information and said viewpoint information, wherein the graphics include visual representations of said status information and controls;

a display that displays the generated graphics such that the graphics are superimposed on the real-time view, wherein the graphics appear attached to the apparatus; and

an interaction device that activates a displayed control.

16. The augmented reality system of claim 15, wherein the apparatus is a robotic device and the interaction device controls the robotic device.