Detection and reconstruction of an environment to facilitate robotic interaction with the environment

• US 9,630,320 B1
• Filed: 07/07/2015
• Issued: 04/25/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Industrial Perception, Inc. (Alphabet Inc.)
• Inventors: Strasdat, Hauke Malte, Konolige, Kurt, Rublee, Ethan, Hinterstoisser, Stefan, Straszheim, Troy, Bradski, Gary
• Primary Examiner(s): Tran, Dalena
• Assistant Examiner(s): Peche, Jorge
• Application Number:
  US14/793,246
• Time in Patent Office: 658 Days
• Field of Search: 700253, 700258, 700259, 901 1, 901 8, 901 46, 901 47
• CPC Class Codes: B25J 19/00 Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators safety-devices in general F16P; protection against radiation in general G21F
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/162 Mobile manipulator, movable base with manipulator arm mounted on it
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 41/008 frames mounted on wheels or caterpillar for movement on rail tracks B65G41/02
• CPC Class Codes: B65G 47/46 and distributing, e.g. automatically, to desired points in tube mail systems B65G51/36
• CPC Class Codes: B65G 47/50 according to destination signals stored in separate systems
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: B65H 67/065 Manipulators with gripping or holding means for transferring the packages from one station to another, e.g. from a conveyor to a creel trolley
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G05B 2219/31312 Identify pallet, bag, box code
• CPC Class Codes: G05B 2219/39391 Visual servoing, track end effector with camera image feedback
• CPC Class Codes: G05B 2219/40053 : Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40298 : Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40442 : Voxel map, 3-D grid map
• CPC Class Codes: G05B 2219/40543 : Identification and location, position of components, objects
• CPC Class Codes: G06T 17/00 : Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/529 : from texture
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/02 : Arm motion controller
• CPC Class Codes: Y10S 901/06 : Communication with another machine
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Methods and systems for detecting and reconstructing environments to facilitate robotic interaction with such environments are described. An example method may involve determining a three-dimensional (3D) virtual environment representative of a physical environment of the robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment. The method may then involve determining two-dimensional (2D) images of the virtual environment including 2D depth maps. The method may then involve determining portions of the 2D images that correspond to a given one or more physical objects. The method may then involve determining, based on the portions and the 2D depth maps, 3D models corresponding to the portions. The method may then involve, based on the 3D models, selecting a physical object from the given one or more physical objects. The method may then involve providing an instruction to the robotic manipulator to move that object.

Claims

1. A method comprising:

determining one or more two-dimensional (2D) images of a plurality of physical objects in a physical environment of a robotic manipulator;

determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects of the plurality, wherein the one or more portions virtually distinguish a boundary of at least one distinct object of the given one or more physical objects from boundaries of other physical objects of the given one or more physical objects;

determining 3D models corresponding to respective determined portions of the one or more 2D images, each 3D model comprising predicted dimensions of a respective physical object of the given one or more physical objects, the predicted dimensions being based on the respective determined portion of the one or more 2D images that corresponds to the respective physical object and further based on dimensions of one or more other physical objects of the plurality;

based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and

providing an instruction to the robotic manipulator to move the particular physical object.

2. The method of claim 1, wherein the one or more other physical objects of the plurality are one or more physical objects of the plurality that were previously moved by the robotic manipulator.

3. The method of claim 1, wherein determining the 3D models comprises determining at least one 3D model having predicted dimensions that fall within a predetermined range of dimensions of the one or more other physical objects, and

wherein the predetermined range of dimensions is bound by one or more of:

a minimum height of the one or more other physical objects, a minimum length of the one or more other physical objects, a minimum width of the one or more other physical objects, an average height of the one or more other physical objects, an average length of the one or more other physical objects, an average width of the one or more other physical objects, a maximum height of the one or more other physical objects, a maximum length of the one or more other physical objects, and a maximum width of the one or more other physical objects.

4. The method of claim 1, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the plurality of physical objects and a reference plane associated with a perspective of one or more sensors, and

wherein determining the 3D models is further based on portions of the 2D depth maps associated with the given one or more physical objects.

5. The method of claim 1, wherein the determined one or more 2D images include one or more 2D orthographic projections of a 3D virtual environment representative of the physical environment.

6. The method of claim 1, wherein respective 3D models of the determined 3D models include two parallel surfaces of an identical shape, wherein the shape is a shape of the respective determined portion of the one or more 2D images to which the respective 3D model corresponds.

7. The method of claim 1, wherein determining the one or more portions of the one or more 2D images comprises determining the one or more portions based on templates corresponding to known polygonic shapes so that the one or more portions include dimensions of the known polygonic shapes.

8. The method of claim 1, further comprising:

refining one or more of the determined 3D models based on templates corresponding to known 3D geometric shapes so that the one or more refined 3D models include dimensions of the known 3D geometric shapes,

wherein selecting the particular physical object from the given one or more physical objects is based on the one or more refined 3D models.

9. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

determining one or more two-dimensional (2D) images of a plurality of physical objects in a physical environment of a robotic manipulator;

determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects of the plurality, wherein the one or more portions virtually distinguish a boundary of at least one distinct object of the given one or more physical objects from boundaries of other physical objects of the given one or more physical objects;

determining 3D models corresponding to respective determined portions of the one or more 2D images, each 3D model comprising predicted dimensions of a respective physical object of the given one or more physical objects, the predicted dimensions being based on the respective determined portion of the one or more 2D images that corresponds to the respective physical object and further based on dimensions of one or more other physical objects of the plurality;

based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and

providing an instruction to the robotic manipulator to move the particular physical object.

10. The non-transitory computer readable medium of claim 9, wherein the plurality of physical objects comprises a pallet of stacked physical objects within the physical environment,

wherein the one or more 2D images include one or more of:

at least one 2D projection associated with a side view of the pallet of stacked physical objects and at least one 2D projection associated with a top-down view of the pallet of stacked physical objects, and

wherein selecting the particular physical object from the given one or more physical objects comprises selecting, as the particular physical object, an object on a top row of the pallet of stacked physical objects closest in proximity to the robotic manipulator.

11. The non-transitory computer readable medium of claim 9, wherein selecting the particular physical object from the given one or more physical objects based on the determined 3D models comprises selecting a physical object corresponding to a 3D model that includes at least three surfaces different from surfaces of the 3D model that are proximate to surfaces of other 3D models of the determined 3D models.

12. The non-transitory computer readable medium of claim 9, the functions further comprising:

based on dimensions of a determined 3D model that corresponds to the particular physical object, determining a location on the particular physical object at which to instruct the robotic manipulator to initiate movement of the particular physical object,

wherein providing the instruction to the robotic manipulator to move the particular physical object comprises providing an instruction to the robotic manipulator to initiate movement of the particular physical object by the robotic manipulator contacting the particular physical object at the determined location on the particular physical object.

13. The non-transitory computer readable medium of claim 9, wherein the one or more other physical objects of the plurality are one or more physical objects of the plurality that were previously moved by the robotic manipulator, the functions further comprising:

determining a range of dimensions of the one or more physical objects of the plurality that were previously moved by the robotic manipulator,

wherein determining the one or more portions of the one or more 2D images is based on the determined range of dimensions of the one or more physical objects of the plurality that were previously moved by the robotic manipulator, and

wherein the predicted dimensions being based on the dimensions of the one or more physical objects of the plurality that were previously moved by the robotic manipulator comprises the predicted dimensions being based on the determined range of dimensions of the one or more physical objects of the plurality that were previously moved by the robotic manipulator.

14. A system comprising:

a robotic manipulator;

at least one processor; and

data storage comprising instructions executable by the at least one processor to cause the system to perform functions comprising;

determining one or more two-dimensional (2D) images of a plurality of physical objects in a physical environment of the robotic manipulator;

determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects of the plurality, wherein the one or more portions virtually distinguish a boundary of at least one distinct object of the given one or more physical objects from boundaries of other physical objects of the given one or more physical objects;

determining 3D models corresponding to respective determined portions of the one or more 2D images, each 3D model comprising predicted dimensions of a respective physical object of the given one or more physical objects, the predicted dimensions being based on the respective determined portion of the one or more 2D images that corresponds to the respective physical object and further based on dimensions of one or more other physical objects of the plurality;

based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and

providing an instruction to the robotic manipulator to move the particular physical object.

15. The system of claim 14, further comprising one or more sensors, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the plurality of physical objects and a reference plane associated with a perspective of the one or more sensors,

wherein at least one edge of a respective determined portion is determined based on boundaries between substantially proximate physical objects, and

wherein the boundaries are indicated by one or more of the 2D depth maps.

16. The system of claim 14, wherein determining the one or more portions of the one or more 2D images comprises determining at least one portion of the one or more 2D images having at least one edge that falls within a predetermined range of dimensions of the plurality of physical objects,

wherein determining the 3D models comprises determining at least one 3D model having predicted dimensions that fall within the predetermined range of dimensions of the plurality of physical objects, and

wherein the predetermined range of dimensions of the plurality of physical objects is bound by one or more of:

a minimum height of the plurality of physical objects, a minimum length of the plurality of physical objects, a minimum width of the plurality of physical objects, an average height of the plurality of physical objects, an average length of the plurality of physical objects, an average width of the plurality of physical objects, a maximum height of the plurality of physical objects, a maximum length of the plurality of physical objects, and a maximum width of the plurality of physical objects.

17. The system of claim 14, wherein determining the 3D models is further based on templates representative of known 3D models, the functions further comprising:

determining one or more features on the 3D model that corresponds to the selected particular physical object based on corresponding marked features in a given template representative of a known 3D model that was used to determine the 3D model that corresponds to the selected particular physical object,

wherein providing the instruction to the robotic manipulator to move the particular physical object comprises providing the instruction to the robotic manipulator to move the particular physical object based on the determined one or more features.

18. The system of claim 14, wherein the plurality of physical objects include cuboid-shaped physical objects, and

wherein determining the one or more portions of the one or more 2D images comprises determining one or more substantially quadrilateral portions of the one or more 2D images.

19. The system of claim 14, further comprising one or more sensors, wherein the physical environment includes one or more walls at least partially surrounding the plurality of physical objects, the functions further comprising:

receiving data from the one or more sensors, the data including a plurality of 3D data points associated with the physical environment; and

determining one or more portions of the 3D data points representative of locations in the physical environment that exceed a threshold distance from the robotic manipulator and are substantially proximate to the one or more walls in the physical environment,

wherein providing the instruction to the robotic manipulator to move the particular physical object is based on the determined one or more portions of the 3D data points and comprises providing an instruction to the robotic manipulator to move the particular physical object while avoiding physical contact between the robotic manipulator and the one or more walls and between the particular physical object and the one or more walls.

20. The system of claim 14, further comprising one or more sensors, wherein the physical environment includes one or more walls at least partially surrounding the plurality of physical objects, the functions further comprising:

receiving data from the one or more sensors, the data including a plurality of 3D data points associated with the physical environment;

determining one or more portions of the 3D data points representative of locations in the physical environment that exceed a threshold distance from the robotic manipulator and are substantially proximate to the one or more walls in the physical environment;

determining one or more 2D orthographic projections of the one or more portions of the 3D data points; and

based on the one or more 2D orthographic projections, determining one or more 3D models associated with respective 2D orthographic projections and representative of the one or more walls in the physical environment,

wherein providing the instruction to the robotic manipulator to move the particular physical object is based on the determined one or more 3D models representative of the one or more walls and comprises providing an instruction to the robotic manipulator to move the particular physical object while avoiding physical contact between the robotic manipulator and the one or more walls and between the particular physical object and the one or more walls.