Vision-guided robots and methods of training them

• US 9,092,698 B2
• Filed: 09/17/2012
• Issued: 07/28/2015
• Est. Priority Date: 06/21/2012
• Status: Active Grant

• Current Assignee: Rethink Robotics
• Original Assignee: Rethink Robotics
• Inventors: Buehler, Christopher J., Siracusa, Michael
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US13/621,658
• Publication Number:
  US 20130345870A1
• Time in Patent Office: 1,044 Days
• Field of Search: 700/245, 700257-259, 706/12, 706/16, 706/20, 706/23, 345629-634, 345/474, 318/568.11, 318/568.16, 318/568.21
• US Class Current: 1/1
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 13/084 Tactile sensors in general G01L5/16, G01L5/22
• CPC Class Codes: B25J 9/0087 Dual arms double SCARA arms B25J9/043
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1682 Dual arm manipulator; Coordination of several manipulators
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/42 Recording and playback systems, i.e. in which the programme is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
• CPC Class Codes: G05B 19/423 Teaching successive positions by walk-through, i.e. the tool head or end effector being grasped and guided directly, with or without servo-assistance, to follow a path
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05B 2219/39466 Hand, gripper, end effector of manipulator
• CPC Class Codes: G05B 2219/40116 Learn by operator observation, symbiosis, show, watch
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 30/1914 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: Y02P 90/80 : Management or planning
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/03 : Teaching system
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/31 : Gripping jaw
• CPC Class Codes: Y10S 901/46 : Sensing device
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Via intuitive interactions with a user, robots may be trained to perform tasks such as visually detecting and identifying physical objects and/or manipulating objects. In some embodiments, training is facilitated by the robot'"'"'s simulation of task-execution using augmented-reality techniques.

Claims

1. A robot-implemented method of learning to complete a specified task involving physical manipulation of an object, the method comprising executing, by the robot, steps comprising:

(a) causing the robot to autonomously perform the task under the control of a robot controller responsive to at least one sensor;

(b) visually displaying, to a user, the autonomous performance of the task by the robot in an augmented-reality display;

(c) delivering, by the user to the robot, performance-altering feedback during the autonomous performance of the task, the robot controller modifying, in response to the feedback, the manner in which the robot continues to autonomously perform the task so the physical manipulation is performed correctly; and

(d) repeating steps (a) through (c) until no further feedback is received.

2. The method of claim 1, wherein steps (a) through (c) are repeated until the robot receives a signal indicative of user approval of the task performance.

3. The method of claim 1, wherein steps (a) through (c) are repeated until a specified time period has elapsed without further performance-altering feedback.

4. The method of claim 1, wherein steps (a) through (c) are repeated until the robot determines, based on an internal success metric, that performance of the task is satisfactory.

5. The method of claim 4, wherein providing an augmented-reality view comprises overlaying graphics onto a camera view captured by the robot.

6. The method of claim 1, wherein the feedback comprises at least one of mechanical or visual feedback.

7. The method of claim 1, wherein the task comprises selecting an object in a camera view.

8. The method of claim 7, wherein visually displaying execution of the task comprises generating an object outline overlaid onto the camera view.

9. The method of claim 7, wherein the feedback comprises an adjustment of the camera view.

10. The method of claim 9, wherein the camera is mounted to an appendage of the robot, the adjustment of the camera view comprising movement of the appendage.

11. The method of claim 7, wherein the feedback comprises movement of the object within a field of view of the camera.

12. The method of claim 1, wherein the task comprises selecting a vision model from among a plurality of computer-implemented vision models and identifying an object in a camera view based on a representation of the object in accordance with the model.

13. The method of claim 12, wherein visually displaying performance of the task comprises overlaying an object outline onto the camera view.

14. The method of claim 12, wherein modifying a manner of performing the task comprises selecting a different vision model.

15. The method of claim 1, wherein the task comprises establishing a grasp point on an object visible in a camera view thereof.

16. The method of claim 15, wherein visually displaying execution of the task comprises overlaying, onto the camera view, a visual representation of an end-effector at the grasp point.

17. The method of claim 16, wherein the feedback comprises a movement of a robot appendage causing a movement of the visual representation of the end-effector in the camera view.

18. A vision-guided robot trainable, via interactions with a human trainer, to complete a specified task involving physical manipulation of an object, the robot comprising:

at least one sensor;

a machine-vision system comprising at least one camera for capturing images within a camera field of view and an image-processing module for identifying objects within the captured images;

a user interface comprising (i) at least one output device for presenting an augmented-reality display visually displaying an autonomous performance of the task by the robot, and (ii) at least one input device for receiving performance-altering feedback from the trainer during the autonomous performance of the task; and

a robot controller, responsive to the at least one sensor and the at least one input device, for controlling autonomous performance of the task by the robot and, in response to the feedback, modifying a manner in which the robot autonomously performs the task until no further feedback is received, whereby the physical manipulation is performed correctly.

19. The robot of claim 18, wherein the at least one output device comprises a monitor for displaying a camera view of the object and robot-generated graphics overlaid thereon.

20. The robot of claim 18, wherein the at least one output device comprises a projector for projecting robot-generated graphics onto the object.

21. The robot of claim 18, further comprising at least one movable appendage for manipulating objects.

22. The robot of claim 21, wherein the at least one movable appendage simultaneously serves as one of the at least one input devices.

23. The robot of claim 18, wherein the image-processing module and the control system comprise sets of instructions stored in memory and executable by a processor.

24. The robot of claim 18, wherein the control system is configured to cause the user interface to display at least one training goal and to receive user feedback regarding the at least one training goal.