Defining and/or applying a planar model for object detection and/or pose estimation

• US 10,102,629 B1
• Filed: 09/10/2015
• Issued: 10/16/2018
• Est. Priority Date: 09/10/2015
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Li, Wen
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Shin, Brian
• Application Number:
  US14/850,913
• Time in Patent Office: 1,132 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods, apparatus, and computer readable storage media related to defining a planar model that approximates a plurality of surfaces of an object and/or applying the planar model to detect the object and/or to estimate a pose for the object. For example, the planar model may be compared to data points sensed by the a three-dimensional vision sensor of a robot to determine the object is present in the field of view of the sensor and/or to determine a pose for the object relative to the robot. A planar model comprises a plurality of planar shapes modeled relative to one another in a three-dimensional space and approximates an object by approximating one or more surfaces of the object with the planar shapes.

Claims

1. A method, comprising:

receiving, from a three-dimensional vision sensor of a robot, a group of data points sensed by the three-dimensional vision sensor;

retrieving, by one or more processors, a three-dimensional planar model that approximates a plurality of surfaces of an object, the planar model defining a plurality of planar shapes modeled relative to one another in a three-dimensional space,

wherein the object is a pallet that includes a plurality of stringers and wherein a first planar shape of the planar model approximates a first stringer surface of a first stringer end of a first stringer of the stringers and a second planar shape of the planar model approximates a second stringer surface of a second stringer end of a second stringer of the stringers, and

wherein the first stringer end and the second stringer end are both on a same side of the pallet;

calculating, by one or more of the processors and for each of a plurality of projections of the model to the data points, differences between the planar model and the data points for a corresponding projection of the projections;

determining, by one or more of the processors, that a projection of the projections satisfies a threshold based on the differences between the planar model and the data points for the projection; and

determining, by one or more of the processors, an object pose for the object relative to the robot based on a planar model pose of the planar model at the projection that satisfies the threshold.

2. The method of claim 1, further comprising adjusting the robot based on the determined object pose.

3. The method of claim 2, wherein the planar model further defines an action area relative to one or more of the planar shapes and wherein adjusting the robot is further based on the action area defined by the planar model.

4. The method of claim 3, wherein the robot includes forks coupled to an elevator, and wherein adjusting the robot includes adjusting a robot pose of the robot to position the forks to encounter the action area.

5. The method of claim 1, wherein the planar model does not approximate any surfaces of the first stringer and the second stringer that are in addition to the first stringer surface and the second stringer surface.

6. The method of claim 1, wherein the pallet further includes a plurality of deckboards coupled to the stringers and wherein a third planar shape of the planar model approximates a deckboard of the deckboards, wherein the deckboard is more proximal to the first stringer end and the second stringer end than any other of the deckboards are to the first stringer end and the second stringer end.

7. The method of claim 6, wherein the planar model does not approximate any surface of at least one of the deckboards.

8. The method of claim 1, wherein a majority of the surfaces modeled by the planar model are modeled by the planar shapes.

9. The method of claim 8, wherein a total quantity of the planar shapes defined by the model is less than fifteen.

10. The method of claim 1, wherein all of the surfaces modeled by the planar model are modeled by the planar shapes.

11. The method of claim 1, wherein the planar model is not computed from any three-dimensional scan data.

12. The method of claim 11, wherein the planar model is generated based on human input provided via one or more user interface input devices.

13. The method of claim 1, wherein the planar shapes are all polygons.

14. The method of claim 13, wherein the polygons are all four-sided.

15. The method of claim 1, further comprising:

translating an origin point of the planar model to a data point of the group of data points; and

generating each projection of a group of the projections based on a different rotation of the planar model when the origin point of the planar model is translated to the data point.

16. A robot, comprising:

at least one non-transitory computer readable storage medium storing a three-dimensional planar model that approximates surfaces of an object, the planar model defining a plurality of planar shapes modeled relative to one another in a three-dimensional space,

wherein the object is a pallet that includes a plurality of stringers and wherein a first planar shape of the planar model approximates a first stringer surface of a first stringer end of a first stringer of the stringers and a second planar shape of the planar model approximates a second stringer surface of a second stringer end of a second stringer of the stringers, and

wherein the first stringer end and the second stringer end are both on a same side of the pallet;

a three-dimensional vision sensor; and

one or more processors in communication with the non-transitory computer readable storage medium and receiving data generated by the three-dimensional vision sensor, wherein the one or more processors are configured to;

receive a group of data points sensed by the three-dimensional vision sensor;

calculate, for each of a plurality of projections of the model to the data points, differences between the planar model and the data points for a corresponding projection of the projections;

determine that a projection of the projections satisfies a threshold based on the differences between the planar model and the data points for the projection;

determine an object pose for the object relative to the robot based on a planar model pose of the planar model at the projection that satisfies the threshold; and

provide, to one or more actuators of the robot, control commands that are based on the determined object pose.

17. The robot of claim 16, wherein the robot includes forks coupled to an elevator, and wherein the control commands adjust a robot pose of the robot to position the forks to encounter fork holes of the pallet.

18. The robot of claim 16, wherein a majority of the surfaces modeled by the planar model are modeled by the planar shapes.

19. The robot of claim 16, wherein all of the surfaces modeled by the planar model are modeled by the planar shapes.

20. The robot of claim 16, wherein the planar model is not computed from any three-dimensional scan data.

21. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

receiving, from a three-dimensional vision sensor of a robot, a group of data points sensed by the three-dimensional vision sensor;

retrieving a three-dimensional planar model that approximates a plurality of surfaces of an object, the planar model defining a plurality of planar shapes modeled relative to one another in a three-dimensional space,

wherein the object is a pallet that includes a plurality of stringers and wherein a first planar shape of the planar model approximates a first stringer surface of a first stringer end of a first stringer of the stringers and a second planar shape of the planar model approximates a second stringer surface of a second stringer end of a second stringer of the stringers, and

wherein the first stringer end and the second stringer end are both on a same side of the pallet;

calculating, for each of a plurality of projections of the model to the data points, differences between the planar model and the data points for a corresponding projection of the projections;

determining that a projection of the projections satisfies a threshold based on the differences between the planar model and the data points for the projection; and

determining an object pose for the object relative to the robot based on a planar model pose of the planar model at the projection that satisfies the threshold.