Pallet detection using units of physical length

• US 9,990,535 B2
• Filed: 04/27/2016
• Issued: 06/05/2018
• Est. Priority Date: 04/27/2016
• Status: Active Grant

• Current Assignee: Crown Equipment Corporation
• Original Assignee: Crown Equipment Corporation
• Inventors: Phillips, Sian, Padhye, Shantanu
• Primary Examiner(s): Couso, Jose
• Application Number:
  US15/139,394
• Publication Number:
  US 20170316253A1
• Time in Patent Office: 769 Days
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20041 Distance transform
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

An image of a physical environment is acquired that comprises a plurality of pixels, each pixel including a two-dimensional pixel location in the image plane and a depth value corresponding to a distance between a region of the physical environment and the image plane. For each pixel, the two dimensional pixel location and the depth value is converted into a corresponding three-dimensional point in the physical environment defined by three coordinate components, each of which has a value in physical units of measurement. A set of edge points is determined within the plurality of three-dimensional points based, at least in part, on the z coordinate component of the plurality of points and a distance map is generated comprising a matrix of cells. For each cell of the distance map, a distance value is assigned representing a distance between the cell and the closest edge point to that cell.

Claims

1. A method of analyzing an image comprising:

acquiring, by a computer, an image of a physical environment, wherein the image comprises a plurality of pixels, each pixel including a two-dimensional pixel location in a plane of the image and a depth value corresponding to a distance between a region of the physical environment and the plane of the image;

for each pixel, converting, by the computer, the two dimensional pixel location and the depth value into a corresponding three-dimensional point in the physical environment, wherein each respective three-dimensional point corresponding to a pixel is defined by three coordinate components (x, y, z), each of which has a value in physical units of measurement;

determining, by the computer, a set of edge points within the plurality of three-dimensional points based, at least in part, on the z coordinate component of one or more of the plurality of points;

generating, by the computer, a distance map, wherein the distance map comprises a matrix of cells;

identifying, by the computer, for each cell of the distance map, a closest one of the edge points;

assigning, by the computer, to each cell of the distance map, a respective distance value representing a distance between the cell and the closest one edge point to that cell; and

based on the respective distance values in the cells of the distance map, determining a location of at least one of a pallet or rack within the physical environment.

2. The method of claim 1, comprising:

locating, by the computer, a first template at a first plurality of different locations on the distance map, wherein at each of the first plurality of different locations, the first template is aligned with a first plurality of cells and has an origin associated with one of the cells; and

at each of the first plurality of different locations, calculating, by the computer, a first corresponding cost function value for the associated one cell.

3. The method of claim 2, wherein the first template is one of a rack template and a pallet template.

4. The method of claim 2, wherein the first template comprises a plurality of template points, each template point is defined by two coordinate components (m, n), each of which has a value in physical units of measurement.

5. The method of claim 4, wherein, for each template point of the first template, its corresponding coordinate component values m, n are each in the same physical units of measurement as the three coordinate components (x, y, z) of the three-dimensional points corresponding to the pixels of the image.

6. The method of claim 2, wherein each first corresponding cost function value is calculated based on a distance between each of the first plurality of cells and their respective closest one edge point.

7. The method of claim 6, wherein the first corresponding cost function is calculated by:

a) multiplying each of the respective distance values from the first plurality of cells with themselves to generate a plurality of squared values; and

b) determining a square root of a mean of the plurality of squared values.

8. The method of claim 6, comprising:

identifying, by the computer, each cell of the distance map having a first corresponding cost function value below a first predetermined threshold.

9. The method of claim 8, comprising:

locating, by the computer, a second template at a second plurality of different locations on the distance map, wherein at each of the second plurality of different locations, the second template is aligned with a second plurality of cells and has an origin associated with one of the cells; and

at each of the second plurality of different locations, calculating, by the computer, a second corresponding cost function value for the associated one cell.

10. The method of claim 9, comprising:

identifying, by the computer, each cell of the distance map having;

a first corresponding cost function value below the first predetermined threshold, and

a second corresponding cost function value below a second predetermined threshold.

11. The method of claim 9, wherein the first template is a rack template and the second template is a pallet template.

12. The method of claim 2, wherein the first template comprises data that identifies the associated first plurality of cells.

13. The method of claim 1, wherein the matrix of cells of the distance map comprise i columns of cells and j rows of cells defining (i×

j) cells, with;

each column referenced by a first index value, a, between 1 and i;

each row referenced by a second index value, b, between 1 and j; and

each cell referenced by an index pair (a, b).

14. The method of claim 13, wherein, for each cell of the distance map, its corresponding index pair values a, b are each in the same physical units of measurement as the three coordinate components (x, y, z) of the three-dimensional points corresponding to the pixels of the image.

15. The method of claim 14, wherein the closest one edge point to the cell of the distance map is determined based on the respective (x, y) coordinate components of each of the edge points and the (a,b) index pair of that cell.

16. The method of claim 15, wherein the distance between the closest one edge point and the cell of the distance map is related to a Euclidian distance between the (x,y) coordinate components of the closest one edge point and the (a,b) index pair of that cell.

17. The method of claim 13, comprising:

identifying, by the computer, a maximum value of the x coordinate component from among all the edge points; and

identifying, by the computer, a maximum value of the y coordinate component from among all the edge points.

18. The method of claim 17, wherein:

i=the maximum value of the x coordinate component; and

j=the maximum value of the y coordinate component.

19. The method of claim 1, wherein determining the set of edge points comprises:

for each of the three-dimensional points;

comparing the z coordinate component of the three-dimensional point with the respective z coordinate component of each three-dimensional point in a predefined neighborhood of the three-dimensional points, to determine an associated edge value; and

determining the three-dimensional point is in the set of edge points when its associated edge value is above a predetermined threshold.

20. The method of claim 1, wherein the respective distance value of each cell in the distance map is in the same physical units of measurement as the three coordinate components (x, y, z) of the points.

21. The method of claim 1, wherein the physical units of measurement is one of centimeters or millimeters.

22. A system for analyzing an image, the system comprising:

a memory device storing executable instructions; and

a processor in communication with the memory device, wherein the processor when executing the executable instructions;

acquires an image of a physical environment, wherein the image comprises a plurality of pixels, each pixel including a two-dimensional pixel location in a plane of the image and a depth value corresponding to a distance between a region of the physical environment and the plane of the image;

for each pixel, converts the two dimensional pixel location and the depth value into a corresponding three-dimensional point in the physical environment, wherein each respective three-dimensional point corresponding to a pixel is defined by three coordinate components (x, y, z), each of which has a value in physical units of measurement;

determines a set of edge points within the plurality of three-dimensional points based, at least in part, on the z coordinate component of one or more of the plurality of points;

generates a distance map, wherein the distance map comprises a matrix of cells;

identifies for each cell of the distance map, a closest one of the edge points;

assigns to each cell of the distance map, a respective distance value representing a distance between the cell and the closest one edge point to that cell; and

determines, based on the respective distance values in the cells of the distance map, a location of at least one of a pallet or rack within the physical environment.

23. The system of claim 22, wherein the processor when executing the executable instructions:

locates a first template at a first plurality of different locations on the distance map, wherein at each of the first plurality of different locations, the first template is aligned with a first plurality of cells and has an origin associated with one of the cells; and

at each of the first plurality of different locations, calculates a first corresponding cost function value for the associated one cell.

24. The system of claim 23, wherein the first template is one of a rack template and a pallet template.

25. The system of claim 23, wherein the first template comprises a plurality of template points, each template point is defined by two coordinate components (m, n), each of which has a value in physical units of measurement.

26. The system of claim 25, wherein, for each template point of the first template, its corresponding coordinate component values m, n are each in the same physical units of measurement as the three coordinate components (x, y, z) of the three-dimensional points corresponding to the pixels of the image.

27. The system of claim 23, wherein each first corresponding cost function value is calculated based on a distance between each of the first plurality of cells and their respective closest one edge point.

28. The system of claim 27, wherein the first corresponding cost function is calculated by:

a) multiplying each of the respective distance values from the first plurality of cells with themselves to generate a plurality of squared values; and

b) determining a square root of a mean of the plurality of squared values.

29. The system of claim 27, wherein the processor when executing the executable instructions:

identifies each cell of the distance map having a first corresponding cost function value below a first predetermined threshold.

30. The system of claim 29, wherein the processor when executing the executable instructions:

locates a second template at a second plurality of different locations on the distance map, wherein at each of the second plurality of different locations, the second template is aligned with a second plurality of cells and has an origin associated with one of the cells; and

at each of the second plurality of different locations, calculates a second corresponding cost function value for the associated one cell.

31. The system of claim 30, wherein the processor when executing the executable instructions:

identifies each cell of the distance map having;

a first corresponding cost function value below the first predetermined threshold, and

a second corresponding cost function value below a second predetermined threshold.

32. The system of claim 30, wherein the first template is a rack template and the second template is a pallet template.

33. The system of claim 23, wherein the first template comprises data that identifies the associated first plurality of cells.

34. The system of claim 22, wherein the matrix of cells of the distance map comprise i columns of cells and j rows of cells defining (i×

j) cells, with;

each column referenced by a first index value, a, between 1 and i;

each row referenced by a second index value, b, between 1 and j; and

each cell referenced by an index pair (a, b).

35. The system of claim 34, wherein, for each cell of the distance map, its corresponding index pair values a, b are each in the same physical units of measurement as the three coordinate components (x, y, z) of the three-dimensional points corresponding to the pixels of the image.

36. The system of claim 35, wherein the closest one edge point to the cell of the distance map is determined based on the respective (x, y) coordinate components of each of the edge points and the (a,b) index pair of that cell.

37. The system of claim 36, wherein the distance between the closest one edge point and the cell of the distance map is related to a Euclidian distance between the (x,y) coordinate components of the closest one edge point and the (a,b) index pair of that cell.

38. The system of claim 34, wherein the processor when executing the executable instructions:

identifies a maximum value of the x coordinate component from among all the edge points; and

identifies a maximum value of the y coordinate component from among all the edge points.

39. The system of claim 38, wherein:

i=the maximum value of the x coordinate component; and

j=the maximum value of the y coordinate component.

40. The system of claim 22, wherein the processor when determining the set of edge points:

for each of the three-dimensional points;

compares the z coordinate component of the three-dimensional point with the respective z coordinate component of each three-dimensional point in a predefined neighborhood of the three-dimensional points, to determine an associated edge value; and

determines the three-dimensional point is in the set of edge points when its associated edge value is above a predetermined threshold.

41. The system of claim 22, wherein the respective distance value of each cell in the distance map is in the same physical units of measurement as the three coordinate components (x, y, z) of the points.

42. The system of claim 22, wherein the physical units of measurement is one of centimeters or millimeters.