Apparatus and method for modifying the operation of a robotic vehicle in a real environment, to emulate the operation of the robotic vehicle operating in a mixed reality environment

• US 8,190,295 B1
• Filed: 04/30/2009
• Issued: 05/29/2012
• Est. Priority Date: 05/14/2008
• Status: Active Grant

• Current Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Original Assignee: Sandia Corporation (Martin Marietta Corporation)
• Inventors: Gladwell, T. Scott, Oppel, Fred J. III, Garretson, Justin R., Parker, Eric P., Rigdon, J. Brian
• Primary Examiner(s): Trammell, James
• Assistant Examiner(s): LANG, MICHAEL DEAN
• Application Number:
  US12/433,347
• Time in Patent Office: 1,125 Days
• Field of Search: 701/23, 701300-302, 700/245, 700/253, 700/255, 700/258, 700/259
• US Class Current: 700/245
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0259 using magnetic or electromagnetic means
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G09B 9/003 for military purposes and tactics
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles

Abstract

Apparatus and methods for modifying the operation of a robotic vehicle in a real environment to emulate the operation of the robotic vehicle in a mixed reality environment include a vehicle sensing system having a communications module attached to the robotic vehicle for communicating operating parameters related to the robotic vehicle in a real environment to a simulation controller for simulating the operation of the robotic vehicle in a mixed (live, virtual and constructive) environment wherein the affects of virtual and constructive entities on the operation of the robotic vehicle (and vice versa) are simulated. These effects are communicated to the vehicle sensing system which generates a modified control command for the robotic vehicle including the effects of virtual and constructive entities, causing the robot in the real environment to behave as if virtual and constructive entities existed in the real environment.

Claims

1. An apparatus for modifying an operation of a robotic vehicle operating in a real environment, to emulate the operation of the robotic vehicle operating in a mixed reality environment, the robotic vehicle including a vehicle control system and a platform, the apparatus comprising:

a vehicle sensing system including a communications module, the vehicle sensing system operatively connected to the vehicle control system and the platform, the vehicle control system operatively arranged to control the operation of the platform in response to an integrated control command, the vehicle sensing system operatively arranged to sense live data corresponding to the live environment within which the robotic vehicle is operating and generate the integrated control command based upon a combination of the live data with virtual data and constructive data;

a simulation controller in communication with the communications module, the simulation controller operatively arranged to receive live data corresponding to the operation of the robotic vehicle in the live environment and simulate the operation of the robotic vehicle in the mixed reality environment, the simulation controller further operatively arranged to simulate actions corresponding to one or more selected from a virtual entity and a constructive entity operating respectively in a virtual environment and a constructive environment, the virtual and the constructive environments corresponding to the live environment, the results of said simulations comprising virtual data corresponding to the actions of virtual entities operating within the virtual environment and constructive data corresponding to the actions of constructive entities operating within the constructive environment, the simulation controller operatively arranged to communicate the virtual data and the constructive data to the vehicle sensing system.

2. The apparatus of claim 1 wherein the robotic vehicle comprises an autonomous robotic vehicle.

3. The apparatus of claim 1 further comprising an operator control input device connected to the simulation processor, the operator control input device operatively arranged to provide operator control data to the vehicle control system and the vehicle control system further operatively arranged to generate the integrated control command based upon a combination of the operator control data, the live data, the virtual data and the constructive data.

4. The apparatus of claim 3 wherein the operator control data comprises a high level task command.

5. The apparatus of claim 1 wherein the vehicle sensing system comprises one or more selected from a visual camera sensor, a magnetic compass sensor, and a global position system (GPS) sensor.

6. The apparatus of claim 1 wherein the virtual data comprises virtual data corresponding to a live entity operating in a second live environment, the second live environment physically separated from the live environment within which the robotic vehicle is operating.

7. The apparatus of claim 1 wherein the vehicle sensing system further comprises a collision detection processor, the collision detection processor operatively arranged to determine the occurrence of collisions between the robotic vehicle operating within the mixed reality environment and the one or more virtual and constructive entities.

8. The apparatus of claim 6 wherein the live entity consists of one or more live entities selected from a second live robotic vehicle entity, a live human-guided vehicle entity, and a live human entity.

9. The apparatus of claim 1 wherein the constructive entity consists of one or more selected from a constructed environmental entity, a constructed human entity and a constructed vehicle entity.

10. The apparatus of claim 1 wherein the vehicle sensing system further comprises a sensor data fusion system, the sensor data fusion system operatively arranged to combine the live data, the virtual data and the constructive data.

11. The apparatus of claim 1 wherein the simulation processor further comprises a memory module for storage of models, the models consisting of one or more models selected from a model representing the robotic vehicle, a model representing a virtual entity and a model representing a constructive entity.

12. A method for modifying an operation of a robotic vehicle operating in a real environment, to emulate the operation of the robotic vehicle operating in a mixed reality environment, the method comprising;

providing a vehicle sensing system operatively connected to the robotic vehicle, the robotic vehicle comprising a vehicle control system and a vehicle platform;

providing a simulation controller in communication with the vehicle sensing system;

receiving by the simulation controller from the vehicle sensing system, live data corresponding to the operation of the robotic vehicle in the live environment, the live data consisting of one or more selected from data corresponding to the live environment within which the robotic vehicle is operating and data corresponding to the operational state of the robotic vehicle;

simulating by the simulation controller, the operation of the robotic vehicle in the mixed reality environment based upon the live data and, simulating in the mixed reality environment actions corresponding to one or more selected from a virtual entity operating in a virtual environment and a constructive entity operating in a constructive environment, the mixed reality environment comprising a combination of the live environment, the virtual environment and the constructive environment, the results of said simulating comprising virtual data corresponding to the actions of the virtual entities and constructive data corresponding to the actions of the constructive entities;

receiving by the vehicle sensing system, the live data corresponding to the live environment within which the robotic vehicle is operating and the data corresponding to the operational state of the robotic vehicle, the virtual data corresponding to the actions of virtual entities operating within the virtual environment and, the constructive data corresponding to the actions of constructive entities operating within the constructive environment; and

,

generating by the vehicle sensing system based upon a combination of the live data, the virtual data and the constructive data, an integrated control command for the robotic vehicle, the combination including one or more interactions within the mixed reality environment of the robotic vehicle with the one or more virtual and constructive entities, the integrated control command controlling a modified operation of the robotic vehicle in the live environment, the modified operation of the robotic vehicle in the live environment thereby emulating the operation of the robotic vehicle operating in the mixed reality environment interacting with the one or more virtual and constructive entities.

13. The method of claim 12 wherein the one or more virtual entities consist of one or more selected from a virtual human entity, and a virtual robotic entity.

14. The method of claim 12 wherein the one or more constructive entities consist of one or more selected from a constructive environmental entity, a constructive human entity, a constructive robotic entity.

15. The method of claim 12 wherein the operation of the robotic vehicle consists of one or more selected from a communications operation and a mobility operation.

16. The method of claim 12 wherein said providing a vehicle sensing system comprises providing a vehicle sensing system including one more live environmental sensors consisting of a visual camera sensor, a magnetic compass sensor, a global positioning system (GPS) sensor, a vehicle speed sensor, and a vehicle attitude sensor.

17. The method of claim 12 wherein providing a simulation controller comprises providing a simulation controller including a memory module for storage of models, the models consisting of one or more models selected from a model representing the robotic vehicle, a model representing a virtual entity and a model representing a constructive entity.

18. The method of claim 12 further comprising the steps of:

providing an operator input device operatively arranged to generate operator control data and communicate the operator control data to the vehicle sensing system; and

, said step of generating further comprises,

generating by the vehicle sensing system based upon a combination of the live data, the virtual data, the constructive data and the operator control data, the integrated control command.

19. The method of claim 12 wherein the modified operation of the robotic vehicle comprises visually tracking an entity in the mixed reality environment.

20. The method of claim 19 wherein the entity consists of one or more selected from a live entity, a virtual entity and a constructive entity.

21. The method of claim 12 wherein the modified operation of the robotic vehicle further comprises emulating a mobility of the robotic vehicle in the mixed reality environment.

22. The method of claim 21 wherein the mobility of the robotic vehicle emulates the robotic vehicle encountering one or more selected from a live environmental obstacle, a virtual environmental obstacle and, a constructive environmental obstacle.