Vision-guided robots and methods of training them

• US 9,701,015 B2
• Filed: 06/24/2015
• Issued: 07/11/2017
• Est. Priority Date: 06/21/2012
• Status: Active Grant

• Current Assignee: Rethink Robotics
• Original Assignee: Rethink Robotics
• Inventors: Buehler, Christopher J., Siracusa, Michael
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US14/749,033
• Publication Number:
  US 20150290803A1
• Time in Patent Office: 748 Days
• Field of Search: 700257, 700245, 700258, 700259, 901 3, 901 31, 901 47, 901 46, 345629, 382155
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 13/084 Tactile sensors in general G01L5/16, G01L5/22
• CPC Class Codes: B25J 9/0087 Dual arms double SCARA arms B25J9/043
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1682 Dual arm manipulator; Coordination of several manipulators
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/42 Recording and playback systems, i.e. in which the programme is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
• CPC Class Codes: G05B 19/423 Teaching successive positions by walk-through, i.e. the tool head or end effector being grasped and guided directly, with or without servo-assistance, to follow a path
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05B 2219/39466 Hand, gripper, end effector of manipulator
• CPC Class Codes: G05B 2219/40116 Learn by operator observation, symbiosis, show, watch
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 30/1914 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: Y02P 90/80 : Management or planning
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/03 : Teaching system
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/31 : Gripping jaw
• CPC Class Codes: Y10S 901/46 : Sensing device
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Via intuitive interactions with a user, robots may be trained to perform tasks such as visually detecting and identifying physical objects and/or manipulating objects. In some embodiments, training is facilitated by the robot'"'"'s simulation of task-execution using augmented-reality techniques.

Claims

1. A robot-implemented method of manipulating objects based on visual recognition thereof, the method comprising causing a robot to execute steps comprising:

(a) selecting an object in a camera field of view;

(b) using a visual model to computationally identify at least a class of the object based on visual detection of general characteristics associated with the object class by prior training;

(c) automatically determining whether an object-specific manipulation routine is stored in association with the stored representation; and

(d) if so, using a controller to execute the object-specific manipulation routine, and if not, using a controller to execute a generic manipulation routine according to the object class.

2. A vision-guided robot for manipulating objects based on visual recognition thereof, the robot comprising:

at least one movable appendage for manipulating objects;

a camera for capturing an image within a camera field of view; and

a controller for (i) computationally identifying at least a class of an object in the image based on visual detection of general characteristics associated with the object class by prior training, (ii) automatically determining whether an object-specific manipulation routine is stored in association with the stored representation, and (iii) if so, causing the at least one movable appendage to execute the object-specific manipulation routine, and if not, causing the at least one movable appendage to execute a generic manipulation routine according to the object class.

3. The robot of claim 2, wherein the control system comprises a data storage medium storing a plurality of visual models and object representations associated therewith.