Robotic vehicle remote control system having a virtual operator environment

• US 8,301,318 B2
• Filed: 03/05/2009
• Issued: 10/30/2012
• Est. Priority Date: 03/05/2008
• Status: Active Grant

• Current Assignee: Robotic Research OPCO LLC
• Original Assignee: Robotic Research LLC
• Inventors: Lacaze, Alberto D., Murphy, Karl N., Smith, Kyle R., Wilhelm, Raymond P. III
• Primary Examiner(s): EL CHANTI, HUSSEIN A
• Application Number:
  US12/398,610
• Publication Number:
  US 20090276105A1
• Time in Patent Office: 1,335 Days
• Field of Search: 701 1- 2, 701/23
• US Class Current: 701/2
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0206 specially adapted to water vehicles

Abstract

A control system for a remotely operated vehicle is disclosed. The control system includes a sensor tracking system configured to sense the remotely operated vehicle. The control system is coupled to the sensor tracking system and is configured to remotely control the remotely operated vehicle. The control system includes a synthetic view generator configured to construct a virtual model of the remotely operated vehicle and its surrounding environment based upon an input from the sensor tracking system. The control system also includes a graphical user interface configured to display a synthetic view of the virtual model. In addition, the control system includes a synthetic viewer control configured to manipulate an orientation of the synthetic view.

Claims

1. A system for controlling a remotely operated vehicle, comprising:

a sensor tracking system configured to sense a remotely operated vehicle, the sensor tracking system being located at a first location and the remotely operated vehicle being located at a second location remote from the first location;

a control system at the first location coupled to the sensor tracking system, the control system configured to remotely control the remotely operated vehicle;

the control system comprising;

a synthetic view generator configured to construct a virtual model of the remotely operated vehicle and its surrounding environment based upon an input from the sensor tracking system;

a graphical user interface configured to display a synthetic view of the virtual model;

a synthetic viewer control configured to manipulate an orientation of the synthetic view; and

an operator control unit configured to control the remotely operated vehicle.

2. The system of claim 1, further comprising a prediction module configured to predict a future state of the remotely operated vehicle based upon information from the sensor tracking system.

3. The system of claim 1, wherein the synthetic view is generated using a stored CAD drawing of the remotely operated vehicle.

4. The system of claim 1, further comprising an operator aid module configured to generate a recommended course of action based upon information from the sensor tracking system.

5. The system of claim 1, further comprising an operator aid module configured to generate warnings regarding a potential course of action based upon information from the sensor tracking system.

6. The system of claim 1, wherein the sensor tracking system includes one or more of the following sensor systems:

a LADAR, a camera, a GPS device, and an inertial guidance system.

7. The system of claim 1, wherein the remotely operated vehicle is unmanned.

8. The system of claim 1, wherein the remotely operated vehicle is manned.

9. The system of claim 8, wherein an operator on the remotely operated vehicle has an option of exercising at least partial control over an operation of the remotely operated vehicle.

10. The system of claim 1, wherein the virtual model is constructed from a surface mesh generated from information provided by the sensor tracking system.

11. A virtual control environment for remotely operating a robotic vehicle, comprising:

a sensor array that detects the robotic vehicle and its surrounding environment, the sensor array being located at a first location and the robotic vehicle being located at a second location remote from the first location;

a module configured to generate a computer model of the robotic vehicle and its surrounding environment based upon information collected by the sensor array;

a synthetic view generator module configured to create virtual views of the robotic vehicle and its surrounding environment based upon the computer model;

a graphical user interface configured to display a virtual view created by the synthetic view generator module; and

an operator control unit located at the first location and configured to control the robotic vehicle based upon the virtual view.

12. The virtual control environment of claim 11, further comprising a prediction module configured to predict a future state of the remotely operated vehicle based upon information from the sensor array.

13. The virtual control environment of claim 11, wherein the virtual view is generated using a stored CAD drawing of the remotely operated vehicle.

14. The virtual control environment of claim 11, further comprising an operator aid module configured to generate a recommended course of action based upon information from the sensor array.

15. The virtual control environment of claim 11, further comprising an operator aid module configured to generate warnings regarding a potential course of action based upon information from the sensor array.

16. The virtual control environment of claim 11, wherein the sensor array includes one or more of the following sensor systems:

a LADAR, a camera, a GPS device, and an inertial guidance system.

17. The virtual control environment of claim 11, wherein the robotic vehicle is unmanned.

18. The virtual control environment of claim 11, wherein the robotic vehicle is manned.

19. The virtual control environment of claim 18, wherein an operator on the robotic vehicle has an option of exercising at least partial control over an operation of the robotic vehicle.

20. The virtual control environment of claim 11, wherein the computer model is constructed from a surface mesh generated from information provided by the sensor array.

21. A method of controlling a remotely operated vehicle with a virtual images of the remotely operated vehicle, comprising:

remotely sensing the remotely operated vehicle and its surrounding environment with a sensor array, the sensor array being located at a first location and the remotely operated vehicle being located at a second location remote from the first location;

generating a virtual model of the remotely operated vehicle and its surrounding environment based upon information gathered by the sensor array;

generating a virtual view of the remotely operated vehicle and its surrounding environment;

displaying the virtual view on a graphical user interface; and

controlling, using a controller, the operation of the remotely operated vehicle based upon viewing the virtual view, the controller being located at the first location.

22. The method of claim 21, further comprising predicting a future state of the remotely operated vehicle based upon information from the sensor array.

23. The method of claim 21, further comprising displaying a live image of the remotely operated vehicle on the graphical user interface.

24. The method of claim 21, further comprising generating a recommended course of action based upon information from the sensor array.

25. The method of claim 21, further comprising generating warnings regarding a potential course of action based upon information from the sensor array.