Object pickup strategies for a robotic device

• US 10,518,410 B2
• Filed: 05/01/2018
• Issued: 12/31/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Bradski, Gary, Croft, Steve, Konolige, Kurt, Rublee, Ethan, Straszheim, Troy, Zevenbergen, John, Hinterstoisser, Stefan, Strasdat, Hauke
• Primary Examiner(s): Tran, Khoi H
• Assistant Examiner(s): Peche, Jorge O
• Application Number:
  US15/968,323
• Publication Number:
  US 20180243904A1
• Time in Patent Office: 609 Days
• CPC Class Codes: B25J 19/00 Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators safety-devices in general F16P; protection against radiation in general G21F
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/162 Mobile manipulator, movable base with manipulator arm mounted on it
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 41/008 frames mounted on wheels or caterpillar for movement on rail tracks B65G41/02
• CPC Class Codes: B65G 47/46 and distributing, e.g. automatically, to desired points in tube mail systems B65G51/36
• CPC Class Codes: B65G 47/50 according to destination signals stored in separate systems
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: B65H 67/065 Manipulators with gripping or holding means for transferring the packages from one station to another, e.g. from a conveyor to a creel trolley
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G05B 2219/31312 Identify pallet, bag, box code
• CPC Class Codes: G05B 2219/39391 Visual servoing, track end effector with camera image feedback
• CPC Class Codes: G05B 2219/40053 : Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40298 : Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40442 : Voxel map, 3-D grid map
• CPC Class Codes: G05B 2219/40543 : Identification and location, position of components, objects
• CPC Class Codes: G06T 17/00 : Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/529 : from texture
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/02 : Arm motion controller
• CPC Class Codes: Y10S 901/06 : Communication with another machine
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Example embodiments may relate to methods and systems for selecting a grasp point on an object. In particular, a robotic manipulator may identify characteristics of a physical object within a physical environment. Based on the identified characteristics, the robotic manipulator may determine potential grasp points on the physical object corresponding to points at which a gripper attached to the robotic manipulator is operable to grip the physical object. Subsequently, the robotic manipulator may determine a motion path for the gripper to follow in order to move the physical object to a drop-off location for the physical object and then select a grasp point, from the potential grasp points, based on the determined motion path. After selecting the grasp point, the robotic manipulator may grip the physical object at the selected grasp point with the gripper and move the physical object through the determined motion path to the drop-off location.

Claims

1. A method comprising:

determining, by a computing device, two or more potential grasp points on a physical object corresponding to points at which a gripper attached to a robotic manipulator is operable to grip the physical object;

determining a target placement of the physical object at a drop-off location;

based at least on the target placement of the physical object at the drop-off location, selecting a grasp point from among the two or more potential grasp points; and

providing instructions to cause the robotic manipulator to grip the physical object at the selected grasp point with the gripper and move the physical object to the drop-off location.

2. The method of claim 1, wherein determining the target placement comprises determining a target positioning and orientation of the physical object relative to one or more other objects at the drop-off location.

3. The method of claim 1, wherein determining the target placement comprises determining a target positioning and orientation of the physical object within the drop-off location.

4. The method of claim 1, further comprising:

determining geometric characteristics of the physical object and geometric characteristics of the drop-off location,

wherein determining the target placement is based at least on the geometric characteristics of the physical object and on the geometric characteristics of the drop-off location.

5. The method of claim 4, wherein the geometric characteristics of the physical object comprise a first size and a first shape, and wherein the geometric characteristics of the drop-off location comprise a second size and a second shape.

6. The method of claim 5, wherein determining the target placement based at least on the geometric characteristics of the physical object and on the geometric characteristics of the drop-off location comprises:

determining the target placement based on the target placement enabling a fitting of the physical object having the first size and first shape within the drop-off location having the second size and second shape.

7. The method of claim 5, wherein another object is placed or expected to be placed at the drop-off location, wherein the other object has a third size and a third shape, and wherein determining the target placement comprises:

determining the target placement based on the target placement enabling a fitting, within the drop-off location having the second size and second shape, of both (i) the physical object having the first size and first shape and (ii) the other object having the third size and third shape.

8. The method of claim 1, wherein selecting the grasp point based at least on the target placement comprises:

selecting a given grasp point, from among the two or more potential grasp points, based on the given grasp point successfully enabling the target placement of the physical object at the drop-off location by the robotic manipulator.

9. The method of claim 1, wherein the provided instructions also cause the robotic manipulator to place the physical object at the drop-off location in accordance with the target placement.

10. The method of claim 1, wherein selecting the grasp point is further based on the selected grasp point minimizing a cost function, and wherein the cost function comprises a plurality of weighted criterion associated with the grasp point.

11. The method of claim 10, wherein the plurality of weighted criterion comprise one or more of:

(i) a spatial location of the grasp point in physical space, (ii) an expected positioning of the robotic manipulator relative to another object when gripping at the grasp point, (iii) an expected configuration of the robotic manipulator when approaching the physical object for gripping at the grasp point, (iv) an expected configuration of the robotic manipulator when moving the physical object while gripping at the grasp point, and (v) an expected configuration of the robotic manipulator when placing the physical object at the drop-off location while gripping at the grasp point.

12. A system comprising:

a robotic manipulator;

a gripper attached to the robotic manipulator; and

a control system configured to;

determine two or more potential grasp points on a physical object corresponding to points at which the gripper is operable to grip the physical object;

determine a target placement of the physical object at a drop-off location;

based at least on the target placement of the physical object at the drop-off location, select a grasp point from among the two or more potential grasp points; and

provide instructions to cause the robotic manipulator to grip the physical object at the selected grasp point with the gripper and move the physical object to the drop-off location.

13. The system of claim 12, wherein the control system is configured to determine the target placement by determining a target positioning and orientation of the physical object relative to one or more other objects at the drop-off location.

14. The system of claim 12, wherein the control system is configured to determine the target placement by determining a target positioning and orientation of the physical object within the drop-off location.

15. The system of claim 12, wherein the control system is further configured to:

determine geometric characteristics of the physical object and geometric characteristics of the drop-off location,

wherein determining the target placement is based at least on the geometric characteristics of the physical object and on the geometric characteristics of the drop-off location.

16. The system of claim 12, wherein the control system being configured to select the grasp point based at least on the target placement comprises the control system being configured to:

select a given grasp point, from among the two or more potential grasp points, based on the given grasp point successfully enabling the target placement of the physical object at the drop-off location by the robotic manipulator.

17. The system of claim 12, wherein the provided instructions also cause the robotic manipulator to place the physical object at the drop-off location in accordance with the target placement.

18. The system of claim 12, wherein the control system is further configured to select the grasp point based on the selected grasp point minimizing a cost function, and wherein the cost function comprises a plurality of weighted criterion associated with the grasp point.

19. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

determining two or more potential grasp points on a physical object corresponding to points at which a gripper attached to a robotic manipulator is operable to grip the physical object;

determining a target placement of the physical object at a drop-off location;

based at least on the target placement of the physical object at the drop-off location, selecting a grasp point from among the two or more potential grasp points; and

providing instructions to cause the robotic manipulator to grip the physical object at the selected grasp point with the gripper and move the physical object to the drop-off location.

20. The non-transitory computer readable medium of claim 19, the functions further comprising:

determining geometric characteristics of the physical object and geometric characteristics of the drop-off location,

wherein determining the target placement is based at least on the geometric characteristics of the physical object and on the geometric characteristics of the drop-off location.