TELEOPERATION METHOD AND HUMAN ROBOT INTERFACE FOR REMOTE CONTROL OF A MACHINE BY A HUMAN OPERATOR

• US 20120004791A1
• Filed: 03/17/2009
• Published: 01/05/2012
• Est. Priority Date: 03/17/2009
• Status: Active Grant

• Current Assignee: Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Original Assignee: Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Inventors: Buelthoff, Heinrich H., Giordano, Paolo Robuffo
• Granted Patent: US 8,634,969 B2
• Time in Patent Office: Days
• US Class Current: 701/2
• CPC Class Codes: G05D 1/005 by providing the operator with signals other than visual, e.g. acoustic, haptic
• CPC Class Codes: G09B 9/02 for teaching control of vehicles or other craft
• CPC Class Codes: G09B 9/48 a model being viewed and manoeuvred from a remote point

Abstract

The invention relates to a teleoperation method and a human robot interface for remote control of a machine by a human operator (5) using a remote control unit, particularly for remote control of a drone, wherein a vestibular feedback is provided to the operator (5) to enhance the situational awareness of the operator (5), wherein the vestibular feedback represents a real motion of the remote-controlled machine.

Claims

1. Teleoperation method for remote control of a machine by a human operator using a remote control unit, wherein a vestibular feedback is provided to the operator to enhance a situational awareness of the operator, wherein the vestibular feedback represents a real motion of the remote-controlled machine.

2. Teleoperation method according to claim 1, comprising the following steps:

a) Determining vestibular data representing the motion of the remote-controlled machine,

b) Determining the vestibular feedback on the basis of the vestibular data of the remote-controlled machine,

c) Subjecting the operator to a motion corresponding to the motion of the remote-controlled machine, wherein the motion of the operator provides the vestibular feedback and is generated according to the vestibular data of the remote-controlled machine.

3. Teleoperation method according to claim 2, further comprising the following steps for determining the vestibular data:

a) Measuring the vestibular data by onboard sensors of the remote-controlled machine, and

b) Transmitting the measured vestibular data from the remote-controlled machine to the remote control unit for providing the corresponding vestibular feedback to the operator.

4. Teleoperation method according to claim 3, wherein the onboard sensors are selected from a group consisting of:

a) Gyroscopes, and

b) Accelerometers.

5. Teleoperation method according to claim 2, further comprising the following step for determining the vestibular data:

Remote measuring the vestibular data of the remote-controlled machine by a motion tracking system, so that it is not necessary to transmit the vestibular data from the remote-controlled machine to the remote control unit.

6. Teleoperation method according to claim 1, wherein the vestibular feedback is provided to the operator by a robot carrying the operator.

7. Teleoperation method according to claim 6, wherein the robot is a machine selected from a group consisting of:

a) a multi-axis serial manipulator comprising several links connected by joints,

b) a parallel manipulator, and

c) a Stewart platform.

8. Teleoperation method according to claim 1, comprising the following step:

Providing a visual feedback to the operator, wherein the visual feedback represents the motion of the remote-controlled machine.

9. Teleoperation method according to claim 8, comprising the following steps for providing the visual feedback to the operator:

a) Calculating a virtual scenery on the basis of the vestibular data, and

b) Displaying the virtual scenery to the operator.

10. Teleoperation method according to claim 8, comprising the following steps for providing the visual feedback to the operator:

a) Generating images from a perspective of the remote-controlled machine by means of an onboard camera,

b) Transmitting the images from the remote-controlled machine to the control unit, and

c) Displaying the images to the operator.

11. Teleoperation method according to claim 1, wherein the vestibular data comprise the following data:

a) Attitude of the remote-controlled machine, namely pitch angle, roll angle and yaw angle of the remote-controlled machine,

b) Speed of the remote-controlled machine, namely temporal changes of the pitch angle, the roll angle and/of the yaw angle of the remote-controlled machine, and

c) Accelerations of the remote-controlled machine, namely accelerations of the pitch angle, the roll angle, the yaw angle and/or the linear coordinates of the remote-controlled machine.

12. Teleoperation method according to claim 1, wherein the remote-controlled machine is selected from a group consisting of:

a) an unmanned aerial vehicle,

b) an unmanned ground vehicle,

c) an underwater vehicle,

d) a nanorobot which is adapted to move through the body of an animal being,

e) a spacecraft, and

f) an unmanned ground combat vehicle.

13. Teleoperation method according to claim 1, further comprising the following steps:

a) Filtering the vestibular data,

b) Determining the vestibular feedback on the basis of the filtered vestibular data, wherein the filtering of the vestibular data ensures that the vestibular feedback complies with safety restrictions and mechanical restrictions of a robot carrying the operator.

14. Teleoperation method according to claim 1, wherein a multi-sensory feedback is provided to the operator comprising the following types of feedback in addition to the vestibular feedback:

a) a visual feedback, wherein the visual feedback preferably represents a perspective of the remote-controlled machine,

b) an acoustical feedback, and

c) a haptic and force feedback.

24. Teleoperation method according to claim 1, further comprising the following steps:

a) Generating motion commands at the remote control unit, wherein the motion commands define a desired motion of the remote-controlled machine,

b) Transmission of the motion commands from the remote control unit to the remote-controlled machine, and

c) Deriving the vestibular data from the motion commands which are sent from the remote control unit to the remote-controlled machine.

15. Human robot interface for remote control of a remote-controlled machine by a human operator using a remote control unit, wherein the human robot interface provides a vestibular feedback to the operator to enhance the situational awareness of the operator, wherein the vestibular feedback represents a motion of the remote-controlled machine.

16. Human robot interface according to claim 15, comprising

a) a receiver for receiving vestibular data from the remote-controlled machine, where the vestibular data are determined by onboard sensors,

b) a controller for determining the vestibular feedback on the basis of the vestibular data received from the remote-controlled machine,

c) a vestibular feedback device for providing the vestibular feedback to the operator.

17. Human robot interface according to claim 15, comprising a robot for providing the vestibular feedback to the operator, wherein the robot carries the operator.

18. Human robot interface according to claim 17, wherein the robot is selected from the group consisting of:

a) a serial manipulator comprising several links connected by joints,

b) a parallel manipulator, and

c) a Stewart platform.

19. Human robot interface according to claim 17, wherein the robot carries at least one of the following components:

a) A visual feedback device,

b) an acoustic feedback device providing an acoustic feedback to the operator,

c) a haptic feedback device and a force feedback device providing a haptic feedback to the operator,

d) a seat for the operator, and

e) an input device for controlling the motion of the remote-controlled machine.

20. Human robot interface according to claim 16, comprising a wireless link between the remote-controlled machine and the remote control unit for transmitting the vestibular data from the remote-controlled machine to the remote control unit and for transmitting motion commands from the remote control unit to the remote-controlled machine.

21. Human robot interface according to claim 15, wherein the remote-controlled machine is selected from a group consisting of:

a) an unmanned aerial vehicle,

b) an unmanned ground vehicle,

c) an underwater vehicle,

d) a nanorobot which is adapted to move through a body of an animal being, and

e) a spacecraft.

22. A method of using the human robot interface according to claim 15 for one of the following purposes:

a) Training of pilots of vehicles,

b) Remote control of unmanned vehicles,

c) Remote control of a swarm of unmanned aerial vehicles, and

d) Navigating of a nanorobot through a body of a human being.

23. Human robot interface according to claim 15, wherein

a) the remote control unit comprises an input device, so that the operator can input motion commands at the input device, wherein the motion commands define a desired motion of the remote-controlled machine,

b) the remote control unit comprises a transceiver adapted for transmitting the motion commands from the remote control unit to the remote-controlled machine, and

c) the remote control unit comprises a processing unit adapted for deriving the vestibular data from the motion commands which are sent from the remote control unit to the remote-controlled machine.