Methods and systems for detecting, recognizing, and localizing pallets

• US 10,328,578 B2
• Filed: 04/21/2017
• Issued: 06/25/2019
• Est. Priority Date: 04/21/2017
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Holz, Dirk
• Primary Examiner(s): Goradia, Shefali D
• Application Number:
  US15/494,227
• Publication Number:
  US 20180304468A1
• Time in Patent Office: 795 Days
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B66F 9/00 Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes mobile or transportable lifts in, or associated with, buildings and specially adapted to be shifted from one part of a building or other structure to another part or to another building or structure B66B9/16; cranes B66C
• CPC Class Codes: B66F 9/063 Automatically guided
• CPC Class Codes: B66F 9/0755 Position control; Position detectors
• CPC Class Codes: G05B 19/402 characterised by control arrangements for positioning, e.g. centring a tool relative to a hole in the workpiece, additional detection means to correct position G05B19/19 takes precedence
• CPC Class Codes: G05B 2219/37582 Position, angle of workpiece surface
• CPC Class Codes: G05D 1/0225 involving docking at a fixed facility, e.g. base station or loading bay parking aids B62D15/027
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/75 involving models

Abstract

Example implementations may relate methods and systems for detecting, recognizing, and localizing pallets. For instance, a computing system may receive sensor data representing aspects of an environment, and identify a set of edge points in the sensor data. The computing system may further determine a set of line segments from the set of edge points where each line segment may fit to a subset of the set of edge points. Additionally, the computing system may also filter the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template, and identify, from the filtered set of line segments, a subset of line segments that align with the pallet template. Based on the identified subset of line segments, the computing system may determine a pose of a pallet in the environment.

Claims

1. A method comprising:

receiving, at a computing system and from a sensor, sensor data representing aspects of an environment;

identifying a set of edge points in the sensor data, wherein identifying the set of edge points involves;

(i) determining a first local surface normal relative to a surface represented in a cluster of data points in the sensor data and a second local surface normal relative to the surface represented in the cluster of data points, and (ii) identifying data points in the cluster of data points that correspond to edge points based on the first local surface normal and the second local surface normal;

determining a set of line segments from the set of edge points, wherein each line segment is fit to a subset of the set of edge points;

filtering the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template;

identifying, from the filtered set of line segments, a subset of line segments that align with the pallet template; and

determining a pose of a pallet in the environment based on the identified subset of line segments.

2. The method of claim 1, wherein receiving sensor data representing aspects of the environment comprises:

receiving depth images representing aspects of the environment from a depth sensor.

3. The method of claim 1, wherein identifying the set of edge points in the sensor data comprises:

determining covariance of data points in a cluster of data points in the sensor data; and

identifying data points in the cluster of data points that correspond to edge points based on the covariance of the data points.

4. The method of claim 1, wherein determining the set of line segments from the set of edge points comprises:

selecting a first edge point and a second edge point from the detected edge points;

estimating a line segment model that includes at least the first edge point and the second edge point;

modifying the line segment model based on an additional edge point located proximate to the line segment model; and

based on the modified line segment model, determining a line segment that includes the first edge point, the second edge point, and the additional edge point.

5. The method of claim 1, further comprising:

causing a robotic device to insert tines into pockets of the pallet; and

responsive to causing the robotic device to insert tines into pockets of the pallet, causing the robotic device to lift the pallet.

6. The method of claim 1, further comprising:

responsive to receiving sensor data representing aspects of the environment, performing a filter process to remove data points of the sensor data that represent surfaces at amplitudes outside a range of amplitudes associated with a pallet positioned on a ground surface.

7. The method of claim 1, wherein filtering the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template comprises:

determining an orientation of a line segment of the set of line segments;

based on the orientation of the line segment, determining whether a length of the line segment is outside either the height range or the width range associated with the dimensions of the pallet template; and

based on determining that the length of the line segment is outside either the height range or the width range, modifying the set of line segments to exclude the line segment.

8. The method of claim 1, wherein identifying, from the filtered set of line segments, the subset of line segments that align with the pallet template comprises:

determining the subset of line segments that align with the dimensions of the pallet template using a registration process.

9. The method of claim 1, further comprising:

determining a confidence level associated with an alignment of the subset of line segments with the dimensions of the pallet template; and

based on the confidence level, determining a control strategy for a robotic device using the pose of the pallet.

10. The method of claim 1, wherein determining the pose of the pallet in the environment based on the identified subset of line segments comprises:

determining a distance and a relative position of the pallet from the sensor; and

determining an orientation of the pallet relative to the sensor.

11. A robotic device, comprising:

a sensor coupled to the robotic device; and

a control system configured to;

receive sensor data representing aspects of an environment, wherein the sensor data includes depth measurements of the environment from a three-dimensional (3D) camera;

identify a set of edge points in the sensor data, wherein identifying the set of edge points in the sensor data involves;

(i) determining a cluster of local neighborhood data points in an depth image from the 3D camera, (ii) determining depth measurements of data points in the cluster of local neighborhood data points, and (iii) identifying data points of the cluster of local neighborhood data points that correspond to edge points based on the determined depth measurements;

determine a set of line segments from the set of edge points, wherein each line segment is fit to a subset of the set of edge points;

filter the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template;

identify, from the filtered set of line segments, a subset of line segments that align with the pallet template; and

determine a pose of a pallet in the environment based on the identified subset of line segments.

12. The system of claim 11, wherein the robotic device is an autonomous robotic forklift.

13. The system of claim 11, wherein the sensor data includes a color image from a set of stereo cameras, and

wherein the control system is configured to convert the color image into a grayscale image to identify the set of edge points in the sensor data.

14. The system of claim 11, wherein the sensor data includes a depth image from a 3D camera, and

wherein the control system is further configured to:

determine a cluster of local neighborhood data points in the depth image;

determine depth measurements corresponding to data points in the cluster of local neighborhood data points; and

identify data points of the cluster of local neighborhood data points that correspond to edge points based on the determined depth measurements.

15. A non-transitory computer readable medium having stored therein program instructions executable by a computing system to cause the computing system to perform operations, the operations comprising:

receiving sensor data representing aspects of an environment;

identifying a set of edge points in the sensor data, wherein identifying the set of edge points involves;

(i) determining a first local surface normal relative to a surface represented in a cluster of data points in the sensor data and a second local surface normal relative to the surface represented in the cluster of data points, and (ii) identifying data points in the cluster of data points that correspond to edge points based on the first local surface normal and the second local surface normal;

determining a set of line segments from the set of edge points, wherein each line segment is fit to a subset of the set of edge points;

filtering the set of line segments to exclude line segments that have a length outside a height range and a width range associated with dimensions of a pallet template;

identifying, from the filtered set of line segments, a subset of line segments that align with the pallet template; and

determining a pose of a pallet in the environment based on the identified subset of line segments.

16. The non-transitory computer readable medium of claim 15, the operations further comprising:

causing a robotic device to lift the pallet and modify a position of the pallet.

17. The non-transitory computer readable medium of claim 15, wherein the width range is the width of the pallet template plus or minus a fixed error tolerance.

18. The non-transitory computer readable medium of claim 15, wherein the height range is the height of the pallet template plus or minus a fixed error tolerance.