Human-Robot Interface Apparatuses and Methods of Controlling Robots

• US 20120072023A1
• Filed: 09/22/2010
• Published: 03/22/2012
• Est. Priority Date: 09/22/2010
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Ota, Yasuhiro
• Granted Patent: US 8,781,629 B2
• Time in Patent Office: Days
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246

Abstract

A method of controlling a robot using a human-robot interface apparatus in two-way wireless communication with the robot includes displaying on a display interface a two-dimensional image, an object recognition support tool library, and an action support tool library. The method further includes receiving a selected object image representing a target object, comparing the selected object image with a plurality of registered object shape patterns, and automatically recognizing a registered object shape pattern associated with the target object if the target object is registered with the human-robot interface. The registered object shape pattern may be displayed on the display interface, and a selected object manipulation pattern selected from the action support tool library may be received. Control signals may be transmitted to the robot from the human-robot interface. Embodiments may also include human-robot apparatuses (HRI) programmed to remotely control a robot.

Claims

1. A method of controlling a robot using a human-robot interface apparatus in two-way wireless communication with the robot, the method comprising:

displaying a two-dimensional image on a display interface of the human-robot interface apparatus, wherein the two-dimensional image is provided by two video cameras associated with the robot;

displaying on the display interface a visual representation of an object recognition support tool library and a visual representation of an action support tool library;

receiving a selected object image selected from the two-dimensional image, wherein the selected object image represents a target object;

comparing the selected object image with a plurality of registered object shape patterns and automatically recognizing a registered object shape pattern associated with the target object if the target object is registered with the human-robot interface apparatus;

displaying the registered object shape pattern on the display interface;

receiving a selected object manipulation pattern selected from the action support tool library; and

transmitting control signals to the robot from the human-robot interface apparatus, wherein the control signals cause the robot to automatically approach the target object and perform an object manipulation of the target object in accordance with the selected object manipulation pattern.

2. The method of claim 1, wherein the selected object image corresponds to a specified portion of the two-dimensional image that includes the selected object image, and the specified portion is selected by a user of the human-robot interface apparatus.

3. The method of claim 1, wherein the registered object shape pattern is automatically recognized using a feature-based matching algorithm.

4. The method of claim 1, wherein the registered object shape pattern is automatically recognized using a speeded-up robust features algorithm.

5. The method of claim 1, wherein:

the visual representation of the object recognition support tool library displays a plurality of shapes; and

the method further comprises registering an object shape pattern associated with the target object if a registered object shape pattern corresponding to the selected object image is not automatically recognized by receiving a selected shape from a plurality of shapes, the selected shape being most similar to a shape of the target object.

6. The method of claim 5, wherein the selected shape is selected from the object recognition support tool library by a user of the human-robot interface apparatus.

7. The method of claim 1, wherein:

each registered object shape pattern has one or more associated object manipulation patterns; and

each object manipulation pattern comprises instructions operable to be provided to the robot such that the robot executes the object manipulation.

8. The method of claim 7, wherein each object manipulation pattern instructs the robot where to grasp the target object and how to move the target object.

9. A human-robot interface apparatus comprising:

a processor;

a wireless communications module, wherein the wireless communications module is in two-way wireless communication with a robot, the robot comprising two video cameras, a multi-fingered manipulator, and a locomotion base;

a computer readable memory comprising an object recognition support tool library, an action support tool library, and a set of tangibly embodied executable instructions, wherein registered object shape patterns corresponding to a plurality of registered objects are stored in the object recognition support tool library, and object manipulation patterns corresponding to object manipulations of the plurality of registered objects are stored in the action support tool library; and

a display interface for displaying a two-dimensional image provided by the two video cameras, a visual representation of the object recognition support tool library, and a visual representation of the action support tool library,

wherein the set of executable instructions, when executed by the processor, causes the human-robot interface apparatus to;

receive selected object data corresponding to a selected object image displayed within the two-dimensional image presented on the display interface, wherein the selected object image represents a target object;

automatically recognize a registered object shape pattern corresponding to the selected object data and the selected object image based at least in part on the plurality of registered objects stored in the object recognition support tool library;

receive an object manipulation pattern selected from the action support tool library; and

wirelessly instruct the robot via the wireless communications module to automatically approach the target object corresponding to the selected object image using the locomotion base, and to perform an object manipulation on the target object using the multi-fingered manipulator based at least in part on the selected object manipulation pattern.

10. The human-robot interface apparatus of claim 9, wherein the selected object image corresponds to a specified portion of the two-dimensional image that includes the selected object image.

11. The human-robot interface apparatus of claim 9, wherein the registered object shape pattern is automatically recognized using a feature-based matching algorithm.

12. The human-robot interface apparatus of claim 9, wherein the registered object shape pattern is automatically recognized using a speeded-up robust features algorithm.

13. The human-robot interface apparatus of claim 9, wherein:

the visual representation of the object recognition support tool library displays a plurality of shapes; and

if a registered object shape pattern corresponding to the selected object image is not automatically recognized, the set of executable instructions, when executed by the processor, further causes the human-robot interface apparatus to register an object shape pattern associated with the target object by receiving a selected shape from a plurality of shapes, the selected shape being most similar to a shape of the target object.

14. The human-robot interface apparatus of claim 13, wherein the selected shape is selected from the object recognition support tool library by a user of the human-robot interface apparatus.

15. The human-robot interface apparatus of claim 9, wherein:

each registered object shape pattern has one or more associated object manipulation patterns; and

each object manipulation pattern comprises instructions operable to be provided to the robot such that the robot executes the object manipulation.

16. The human-robot interface apparatus of claim 9, wherein the robot automatically approaches the target object using a rapidly-exploring random tree route-planning algorithm.

17. A human-robot interface apparatus comprising:

a wireless communications module, wherein the wireless communications module is in two-way wireless communication with a robot, the robot comprising two video cameras, a multi-fingered manipulator, and a locomotion base;

a computer readable memory comprising an object recognition support tool library and an action support tool library, wherein registered object shape patterns corresponding to a plurality of registered objects are stored in the object recognition support tool library, and object manipulation patterns corresponding to object manipulations of the plurality of registered objects are stored in the action support tool library; and

a display interface for displaying a two-dimensional image provided by the two video cameras, a visual representation of the object recognition support tool library, and a visual representation of the action support tool library,

wherein the human-robot interface apparatus is programmed to;

automatically recognize a registered object shape pattern corresponding to a selected object image using a feature-based matching algorithm, the selected object image being selected from the two-dimensional image and representing a target object;

register an object shape pattern associated with the target object if a registered object shape pattern corresponding to the selected object image is not automatically recognized by receiving a selected shape from a plurality of shapes displayed in the visual representation of the object recognition support tool library, the selected shape being most similar to a shape of the target object; and

wirelessly transmit control signals to the robot via the wireless communications module to perform an object manipulation on the target object using the multi-fingered manipulator based at least in part on a selected object manipulation pattern selected from the visual representation of the action support tool library.

18. The human-robot interface apparatus of claim 17, wherein the feature-based matching algorithm comprises a speeded-up robust features algorithm.

19. The human-robot interface apparatus of claim 17, wherein:

each registered object shape pattern has one or more associated object manipulation patterns; and

each object manipulation pattern comprises instructions operable to be provided to the robot such that the robot executes the object manipulation.

20. The human-robot interface apparatus of claim 17, wherein the robot automatically approaches the target object using a rapidly-exploring random tree route-planning algorithm.