Continuous updating of plan for robotic object manipulation based on received sensor data

• US 9,238,304 B1
• Filed: 03/14/2014
• Issued: 01/19/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: Industrial Perception, Inc.
• Inventors: Bradski, Gary, Konolige, Kurt, Rublee, Ethan, Straszheim, Troy, Strasdat, Hauke, Hinterstoisser, Stefan
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US14/212,994
• Time in Patent Office: 676 Days
• Field of Search: 700245-262, 901 1- 2, 901 6- 11, 901/14, 901 30- 36
• US Class Current: 1/1
• CPC Class Codes: B25J 19/00 Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators safety-devices in general F16P; protection against radiation in general G21F
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/162 Mobile manipulator, movable base with manipulator arm mounted on it
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 41/008 frames mounted on wheels or caterpillar for movement on rail tracks B65G41/02
• CPC Class Codes: B65G 47/46 and distributing, e.g. automatically, to desired points in tube mail systems B65G51/36
• CPC Class Codes: B65G 47/50 according to destination signals stored in separate systems
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: B65H 67/065 Manipulators with gripping or holding means for transferring the packages from one station to another, e.g. from a conveyor to a creel trolley
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G05B 2219/31312 Identify pallet, bag, box code
• CPC Class Codes: G05B 2219/39391 Visual servoing, track end effector with camera image feedback
• CPC Class Codes: G05B 2219/40053 : Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40298 : Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40442 : Voxel map, 3-D grid map
• CPC Class Codes: G05B 2219/40543 : Identification and location, position of components, objects
• CPC Class Codes: G06T 17/00 : Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/529 : from texture
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/02 : Arm motion controller
• CPC Class Codes: Y10S 901/06 : Communication with another machine
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Example systems and methods allow for dynamic updating of a plan to move objects using a robotic device. One example method includes determining a virtual environment by one or more processors based on sensor data received from one or more sensors, the virtual environment representing a physical environment containing a plurality of physical objects, developing a plan, based on the virtual environment, to cause a robotic manipulator to move one or more of the physical objects in the physical environment, causing the robotic manipulator to perform a first action according to the plan, receiving updated sensor data from the one or more sensors after the robotic manipulator performs the first action, modifying the virtual environment based on the updated sensor data, determining one or more modifications to the plan based on the modified virtual environment, and causing the robotic manipulator to perform a second action according to the modified plan.

Claims

1. A method comprising:

determining a virtual environment by one or more processors based on sensor data received from one or more sensors on a robotic manipulator, the virtual environment representing a physical environment containing a plurality of physical objects;

developing a plan, based on the virtual environment, to cause the robotic manipulator to move one or more of the physical objects in the physical environment;

causing the robotic manipulator to pick up at least one object of the plurality of physical objects with the robotic manipulator, move the at least one object to a drop-off location, and drop off the at least one object at the drop-off location according to the plan;

after the robotic manipulator drops off the at least one object at the drop-off location, causing the robotic manipulator to perform a scan of an area of the physical environment vacated by the at least one object by moving the robotic manipulator while receiving updated sensor data from the one or more sensors on the robotic manipulator, wherein the updated sensor data is indicative of at least one other physical object previously obscured by the at least one object;

modifying the virtual environment based on the updated sensor data;

determining one or more modifications to the plan based on the modified virtual environment; and

causing the robotic manipulator to perform another action according to the modified plan.

2. The method of claim 1, further comprising receiving the updated sensor data from the one or more sensors on the robotic manipulator based on continuous scans of the physical environment by the one or more sensors.

3. The method of claim 1, further comprising receiving the updated sensor data from the one or more sensors on the robotic manipulator based on periodic scans of the physical environment by the one or more sensors at a certain time interval.

4. The method of claim 1, wherein:

the plurality of physical objects comprise a plurality of different shapes of objects; and

the virtual environment comprises information indicative of a shape of at least some of the physical objects within the physical environment.

5. The method of claim 1, wherein the virtual environment comprises a facade of some of the physical objects located along a plane within the physical environment, wherein the facade comprises a two-dimensional depth map of distances of the physical objects from the plane.

6. The method of claim 1, further comprising:

causing the robotic manipulator to move into the area vacated by the at least one object to receive the updated sensor data from a point of view of the area of the physical environment vacated by the at least one object after the robotic manipulator drops off the at least one object; and

determining one or more modifications to the plan based on the updated sensor data received from the point of view of the area of the physical environment vacated by the at least one object.

7. The method of claim 1, wherein determining one or more modifications to the plan comprises determining that the robotic manipulator failed to move an object according to the plan while performing an action according to the plan.

8. The method of claim 1, wherein developing the plan comprises determining an ordering of at least some of the physical objects, and wherein the plan indicates to cause the robotic manipulator to move the physical objects from an area of the physical environment in the determined ordering.

9. The method of claim 1, further comprising:

determining an area of the physical environment vacated by moved physical objects; and

causing a moveable cart on which the robotic manipulator is mounted to move into the vacated area in order to position the robotic manipulator to move one or more other objects of the plurality.

10. The method of claim 1, wherein the plan indicates to cause the robotic manipulator to stack the physical objects onto a pallet within the physical environment, wherein the pallet comprises a portable platform capable of supporting a plurality of physical objects.

11. The method of claim 10, wherein the plan indicates to cause the robotic manipulator to stack the physical objects onto the pallet in a configuration that minimizes a total amount of air gaps between the physical objects.

12. The method of claim 10, wherein the plan indicates to cause the robotic manipulator to stack the physical objects onto the pallet into interlaced columns of physical objects, such that portions of certain physical objects are positioned within at least two of the interlaced columns.

13. The method of claim 1, wherein:

the physical environment comprises a plurality of physical objects on a moving conveyer belt;

the modified virtual environment comprises a representation of a current position of the physical objects on the moving conveyer belt; and

the method further comprises determining one or more modifications to the plan based on the current position of the physical objects in the modified virtual environment.

14. The method of claim 1, wherein the virtual environment comprises a segmentation of the plurality of physical objects, and wherein the method further comprises causing the robotic manipulator to perturb the plurality of physical objects to determine the segmentation of the plurality of physical objects based on the sensor data.

15. A system, comprising:

a robotic manipulator;

at least one sensor on the robotic manipulator; and

a control system configured to;

determine a virtual environment based on sensor data received from the at least one sensor on the robotic manipulator, the virtual environment representing a physical environment containing a plurality of physical objects;

develop a plan, based on the virtual environment, to cause the robotic manipulator to move one or more of the physical objects in the physical environment;

cause the robotic manipulator to pick up at least one object of the plurality of physical objects with the robotic manipulator, move the at least one object to a drop-off location, and drop off the at least one object at the drop-off location according to the plan;

after the robotic manipulator drops off the at least one object at the drop-off location, cause the robotic manipulator to perform a scan of an area vacated by the at least one object by moving the robotic manipulator while receiving updated sensor data from the one or more sensors on the robotic manipulator, wherein the updated sensor data is indicative of at least one other physical object previously obscured by the at least one object;

modify the virtual environment based on the updated sensor data;

determine one or more modifications to the plan based on the modified virtual environment; and

cause the robotic manipulator to perform another action according to the modified plan.

16. The system of claim 15, further comprising a moveable cart, wherein the robotic manipulator is mounted on the moveable cart, and wherein the control system is further configured to:

determine an area of the physical environment vacated by removed physical objects; and

cause the moveable cart to move into the vacated area in order to position the robotic manipulator to perform additional scanning of at least one remaining object of the plurality with the at least one sensor on the robotic manipulator.

17. The system of claim 15, wherein the control system is further configured to:

cause the robotic manipulator to move into the area vacated by the at least one physical object to receive the updated sensor data from a point of view of the area of the physical environment vacated by the at least one object after the robotic manipulator drops off the at least one object; and

determine one or more modifications to the plan based on the sensor data received from the point of view of the area of the physical environment vacated by the at least one object.

18. The system of claim 15, wherein the control system is further configured to develop the plan by determining an ordering of at least some of the physical objects, and wherein the plan indicates to cause the robotic manipulator to move the physical objects from an area of the physical environment in the determined ordering.

19. The system of claim 15, wherein the plan indicates to cause the robotic manipulator to stack the physical objects onto a pallet within the physical environment, wherein the pallet comprises a portable platform capable of supporting a plurality of physical objects.

20. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:

determining a virtual environment by one or more processors based on sensor data received from one or more sensors on a robotic manipulator, the virtual environment representing a physical environment containing a plurality of physical objects;

developing a plan, based on the virtual environment, to cause the robotic manipulator to move one or more of the physical objects in the physical environment;

causing the robotic manipulator to pick up at least one object of the plurality of physical objects with the robotic manipulator, move the at least one object to a drop-off location, and drop off the at least one object at the drop-off location according to the plan;

after the robotic manipulator drops off the at least one object at the drop-off location, causing the robotic manipulator to perform a scan of an area vacated by the at least one object by moving the robotic manipulator while receiving updated sensor data from the one or more sensors on the robotic manipulator, wherein the updated sensor data is indicative of at least one other physical object previously obscured by the at least one object;

modifying the virtual environment based on the updated sensor data;

determining one or more modifications to the plan based on the modified virtual environment; and

causing the robotic manipulator to perform another action according to the modified plan.