Teleoperation method and human robot interface for remote control of a machine by a human operator

• US 8,634,969 B2
• Filed: 03/17/2009
• Issued: 01/21/2014
• Est. Priority Date: 03/17/2009
• Status: Active Grant

• Current Assignee: Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Original Assignee: Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Inventors: Buelthoff, Heinrich H., Giordano, Paolo Robuffo
• Primary Examiner(s): FREJD, RUSSELL WARREN
• Application Number:
  US13/256,438
• Publication Number:
  US 20120004791A1
• Time in Patent Office: 1,771 Days
• Field of Search: 701/2, 700/245, 700/264, 434/30, 434/38, 434/44, 434/45, 434/59
• US Class Current: 701/2
• CPC Class Codes: G05D 1/005 by providing the operator with signals other than visual, e.g. acoustic, haptic
• CPC Class Codes: G09B 9/02 for teaching control of vehicles or other craft
• CPC Class Codes: G09B 9/48 a model being viewed and manoeuvred from a remote point

Abstract

The invention relates to a teleoperation method and a human robot interface for remote control of a machine by a human operator (5) using a remote control unit, particularly for remote control of a drone, wherein a vestibular feedback is provided to the operator (5) to enhance the situational awareness of the operator (5), wherein the vestibular feedback represents a real motion of the remote-controlled machine.

Claims

1. Teleoperation method for remote control of a machine by a human operator using a remote control unit, comprising the following steps:

determining vestibular data representing the motion of the machine,

determining a vestibular feedback on the basis of the vestibular data of the machine, wherein the vestibular feedback represents the real motion of the machine,

providing said vestibular feedback to the operator to enhance situational awareness of the operator,

filtering said vestibular data,

determining said vestibular feedback on the basis of said filtered vestibular data, wherein said filtering of said vestibular data ensures that said vestibular feedback complies with safety restrictions or mechanical restrictions of a robot carrying the operator so that a collision of the remote controlled machine with another flying object or with the ground is not reproduced by the vestibular feedback, and

providing said vestibular feedback to the operator by a multi-axis serial manipulator comprising several links connected by joints.

2. The teleoperation method according to claim 1, further comprising the following step:

a) Subjecting the operator to a motion corresponding to the motion of the remote-controlled machine, wherein the motion of the operator provides the vestibular feedback and is generated according to the vestibular data of the remote-controlled machine.

3. Teleoperation method according to claim 2, further comprising the following steps for determining the vestibular data:

a) Measuring the vestibular data by onboard sensors of the remote-controlled machine, and

b) Transmitting the measured vestibular data from the remote-controlled machine to the remote control unit for providing the corresponding vestibular feedback to the operator.

4. Teleoperation method according to claim 3, wherein the onboard sensors are selected from a group consisting of:

a) Gyroscopes, and

b) Accelerometers.

5. Teleoperation method according to claim 2, further comprising the following step for determining the vestibular data:

Remote measuring the vestibular data of the remote-controlled machine by a motion tracking system, so that it is not necessary to transmit the vestibular data from the remote-controlled machine to the remote control unit.

6. Teleoperation method according to claim 1, comprising the following step:

Providing a visual feedback to the operator, wherein the visual feedback represents the motion of the remote-controlled machine.

7. Teleoperation method according to claim 6, comprising the following steps for providing the visual feedback to the operator:

a) Calculating a virtual scenery on the basis of the vestibular data, and

b) Displaying the virtual scenery to the operator.

8. Teleoperation method according to claim 6, comprising the following steps for providing the visual feedback to the operator:

a) Generating images from a perspective of the remote-controlled machine by means of an onboard camera,

b) Transmitting the images from the remote-controlled machine to the control unit, and

c) Displaying the images to the operator.

9. Teleoperation method according to claim 1, wherein the vestibular data comprise the following data:

a) Attitude of the remote-controlled machine, namely pitch angle, roll angle and yaw angle of the remote-controlled machine,

b) Speed of the remote-controlled machine, namely temporal changes of the pitch angle, the roll angle and the yaw angle of the remote-controlled machine, and

c) Accelerations of the remote-controlled machine, namely accelerations of the pitch angle, the roll angle, the yaw angle and the linear coordinates of the remote-controlled machine.

10. Teleoperation method according to claim 1, wherein the remote-controlled machine is selected from a group consisting of:

a) an unmanned aerial vehicle,

b) an unmanned ground vehicle,

c) an underwater vehicle,

d) a nanorobot which is adapted to move through the body of an animal being,

e) a spacecraft, and

f) an unmanned ground combat vehicle.

11. Teleoperation method according to claim 1, wherein a multi-sensory feedback is provided to the operator comprising the following types of feedback in addition to the vestibular feedback:

a) a visual feedback, wherein the visual feedback preferably represents a perspective of the remote-controlled machine,

b) an acoustical feedback, and

c) a haptic and force feedback.

12. Teleoperation method according to claim 1, further comprising the following steps:

a) Generating motion commands at the remote control unit, wherein the motion commands define a desired motion of the remote-controlled machine,

b) Transmission of the motion commands from the remote control unit to the remote-controlled machine, and

c) Deriving the vestibular data from the motion commands which are sent from the remote control unit to the remote-controlled machine.

13. Human robot interface for remote control of a remote-controlled machine by a human operator using a remote control unit, wherein

the human robot interface provides a vestibular feedback to the operator to enhance the situational awareness of the operator,

the vestibular feedback represents a motion of the remote-controlled machine,

the interface comprising

a filter for filtering vestibular data, so that the vestibular feedback is determined on the basis of said filtered vestibular data, wherein the filtering of said vestibular data ensures that said vestibular feedback complies with safety restrictions or mechanical restrictions of a robot carrying the operator so that a collision of the remote controlled machine with another flying object or with the ground is not reproduced by the vestibular feedback, and

a multi-axis serial manipulator for providing the vestibular feedback, wherein the multi-axis serial manipulator comprises several links connected by joints.

14. Human robot interface according to claim 13, comprising

a) a receiver for receiving vestibular data from the remote-controlled machine, where the vestibular data are determined by onboard sensors,

b) a controller for determining the vestibular feedback on the basis of the vestibular data received from the remote-controlled machine,

c) a vestibular feedback device for providing the vestibular feedback to the operator.

15. Human robot interface according to claim 13, wherein the robot carries at least one of the following components:

a) A visual feedback device,

b) an acoustic feedback device providing an acoustic feedback to the operator,

c) a haptic feedback device and a force feedback device providing a haptic feedback to the operator,

d) a seat for the operator, and

e) an input device for controlling the motion of the remote-controlled machine.

16. Human robot interface according to claim 14, comprising a wireless link between the remote-controlled machine and the remote control unit for transmitting the vestibular data from the remote-controlled machine to the remote control unit and for transmitting motion commands from the remote control unit to the remote-controlled machine.

17. Human robot interface according to claim 13, wherein the remote-controlled machine is selected from a group consisting of:

a) an unmanned aerial vehicle,

b) an unmanned ground vehicle,

c) an underwater vehicle, and

d) a spacecraft.

18. A method of using the human robot interface according to claim 13 for one of the following purposes:

a) Training of pilots of vehicles,

b) Remote control of unmanned vehicles,

c) Remote control of a swarm of unmanned aerial vehicles, and

d) Navigating of a nanorobot through a body of a human being.

19. Human robot interface according to claim 13, wherein

a) the remote control unit comprises an input device, so that the operator can input motion commands at the input device, wherein the motion commands define a desired motion of the remote-controlled machine,

b) the remote control unit comprises a transceiver adapted for transmitting the motion commands from the remote control unit to the remote-controlled machine, and

c) the remote control unit comprises a processing unit adapted for deriving the vestibular data from the motion commands which are sent from the remote control unit to the remote-controlled machine.