VISION-GUIDED ROBOTS AND METHODS OF TRAINING THEM

• US 20130343640A1
• Filed: 09/17/2012
• Published: 12/26/2013
• Est. Priority Date: 06/21/2012
• Status: Abandoned Application

• Current Assignee: Rethink Robotics
• Original Assignee: Rethink Robotics
• Inventors: Buehler, Christopher J., Siracusa, Michael R.
• Application Number:
  US13/621,648
• Publication Number:
  US 20130343640A1
• Time in Patent Office: Days
• US Class Current: 382/155
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 13/084 Tactile sensors in general G01L5/16, G01L5/22
• CPC Class Codes: B25J 9/0087 Dual arms double SCARA arms B25J9/043
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1682 Dual arm manipulator; Coordination of several manipulators
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/42 Recording and playback systems, i.e. in which the programme is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
• CPC Class Codes: G05B 19/423 Teaching successive positions by walk-through, i.e. the tool head or end effector being grasped and guided directly, with or without servo-assistance, to follow a path
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05B 2219/39466 Hand, gripper, end effector of manipulator
• CPC Class Codes: G05B 2219/40116 Learn by operator observation, symbiosis, show, watch
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 30/1914 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: Y02P 90/80 : Management or planning
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/03 : Teaching system
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/31 : Gripping jaw
• CPC Class Codes: Y10S 901/46 : Sensing device
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Via intuitive interactions with a user, robots may be trained to perform tasks such as visually detecting and identifying physical objects and/or manipulating objects. In some embodiments, training is facilitated by the robot'"'"'s simulation of task-execution using augmented-reality techniques.

Claims

1. A method of training a robot to complete a specified task comprising selection of an object, the robot having a machine-vision system for capturing images within a camera field of view, the method comprising:

(a) selecting, by the robot, an object in the field of view and displaying the selection on a robot-controlled visual display, the displayed selection including a camera view of the selected object and a robot-generated object outline overlaid thereon;

(b) based on the display, providing feedback to the robot indicating whether the selection is correct; and

(c) repeating steps (a) and (b) until the selection is correct.

2. The method of claim 1, wherein the selection is based on at least one of mechanical or visual input provided by a user.

3. The method of claim 2, further comprising bringing a robot appendage into physical contact with an object of interest so as to provide the mechanical input.

4. The method of claim 2, further comprising bringing an object of interest into the camera field of view so as to provide the visual input.

5. The method of claim 1, wherein the selection is based on a visual model and at least one associated object representation stored in memory.

6. The method of claim 5, wherein the object representation comprises at least one of an image template, coordinates and texture descriptors associated with object keypoints, or parameters associated with an object shape.

7. A vision-guided robot trainable, via interactions with a human trainer, to complete a specified task comprising selection of an object, the robot comprising:

a machine-vision system comprising at least one camera for capturing images within a camera field of view and an image-processing module for identifying objects within the captured images;

a user interface comprising (i) a robot-controlled visual display for displaying the images and, overlaid thereon, robot-generated graphics indicative of a selection of an object by the robot, and (ii) at least one input device for receiving feedback from the trainer indicating whether the selection is correct; and

a control system for controlling and, in response to the feedback, modifying the selection until the selection is correct.

8. The robot of claim 7, wherein the control system comprises memory storing a plurality of visual models and associated object representations.

9. The robot of claim 8, wherein the control system is adapted to control selection of the object based on an object representation associated with the object.

10. The robot of claim 7, further comprising at least one movable appendage for manipulating objects and receiving feedback from the trainer.

11. A robot-implemented method of learning to visually recognize objects, the method comprising:

(a) establishing a visual outline of an object in a camera view thereof;

(b) based at least in part on an analysis of pixels within the outline, selecting a visual model from among a plurality of computer-implemented visual models and generating a representation of the object in accordance with the selected model;

(c) based on the model, identifying the object in a camera view thereof;

(d) receiving user feedback related to the identification, the feedback either confirming or rejecting the identification; and

(e) repeating steps (b) through (d) until the identification is confirmed.

12. The method of claim 11, wherein establishing the visual outline of the object comprises identifying the object based on user input.

13. The method of claim 12, wherein the user input comprises movement of the object, by the user, into a field of view of the camera providing the camera view.

14. The method of claim 12, wherein the user input comprises movement of the object, by the user, within a field of view of the camera providing the camera view.

15. The method of claim 11, wherein establishing the visual outline of the object comprises displaying an outline in the camera view, receiving user feedback related to the outline, and adjusting the outline based on the feedback until the user confirms that the outline is proper.

16. The method of claim 15, wherein the user feedback comprises repositioning of a camera providing the camera view relative to the object.

17. The method of claim 11, wherein establishing the visual outline comprises receiving an outline drawn by the user on a drawing interface.

18. The method of claim 11, further comprising associating non-visual information with the object.

19. The method of claim 18, further comprising soliciting and receiving the non-visual information from the user.

20. The method of claim 11, wherein the representation comprises at least one of (i) parameters associated with a shape, (ii) coordinates and texture descriptors associated with keypoints, or (iii) an image template and associated similarity threshold.

21. The method of claim 11, further comprising recognizing the object as a member of one of a plurality of object classes defined in memory.

22. A vision-guided robot trainable, via interactions with a human trainer, to complete a specified task comprising visual recognition of an object, the robot comprising:

a camera for capturing camera views of the object;

a control system for (i) selecting, based at least in part on an analysis of pixels within a visual outline of the object in a camera view thereof, a visual model from among a plurality of computer-implemented visual models and generating a representation of the object in accordance with the selected model, (ii) identifying the object in a camera view thereof based on the model; and

a user interface comprising (i) a robot-controlled visual display for displaying a camera view of the object and, overlaid thereon, robot-generated graphics indicative of the identification, and (ii) at least one input device for receiving feedback from the trainer indicating whether the identification is correct.

23. The robot of claim 22, wherein the control system is further configured to modify the selection of the visual model based on the feedback.

24. The robot of claim 22, further comprising at least one movable appendage for manipulating objects and receiving feedback from the trainer.