GENERATING A COMPOSITE SCORE FOR A POSSIBLE PALLET IN AN IMAGE SCENE

• US 20130101204A1
• Filed: 10/16/2012
• Published: 04/25/2013
• Est. Priority Date: 10/19/2011
• Status: Active Grant

• Current Assignee: Crown Equipment Corporation
• Original Assignee: Crown Equipment Corporation
• Inventors: Holeva, Lee F., Elston, Edwin R., Seelinger, Michael J., Yoder, John-David S.
• Granted Patent: US 9,082,195 B2
• Time in Patent Office: Days
• US Class Current: 382/141
• CPC Class Codes: B66F 9/122 longitudinally movable
• CPC Class Codes: B66F 9/24 Electrical devices or systems
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A method is provided for evaluating a possible pallet object in a gray scale image including one or more pallets. The method may comprise: generating, by a computer, a first confidence score Score_{LowerLeftCorner} for a lower left corner associated with the possible pallet object; generating, by the computer, a second confidence score Score_{LowerRightCorner} for a lower right corner associated with the possible pallet object; generating, by the computer, a third confidence score Score_{BaseboardLine} for a bottom pallet board line associated with the possible pallet object; generating, by the computer, a fourth confidence score Score_hole for a center stringer associated with the possible pallet object; and calculating, by the computer, a composite score Score_Object based on the first, second, third and fourth confidence scores.

Claims

1. A method for evaluating a possible pallet object in a gray scale image including one or more pallets, comprising:

generating, by a computer, a first confidence score Score_{LowerLeftCorner} for a lower left corner associated with the possible pallet object;

generating, by the computer, a second confidence score Score_{LowerRightCorner} for a lower right corner associated with the possible pallet object;

generating, by the computer, a third confidence score Score_{BaseboardLine} for a bottom pallet board line associated with the possible pallet object;

generating, by the computer, a fourth confidence score Score_hole for a center stringer associated with the possible pallet object; and

calculating, by the computer, a composite score Score_Object based on the first, second, third and fourth confidence scores, wherein the composite score Score_Object represents a likelihood that the possible pallet object corresponds to an actual pallet structure in the gray scale image.

2. The method of claim 1, further comprising:

determining, using the computer, horizontal cross correlations between the gray scale image and a first step-edge template to generate a set of horizontal cross correlation results;

determining, using said computer, vertical cross correlations between the gray scale image and a second step-edge template to generate a set of vertical cross correlation results;

determining, using said computer, a first set of pixels, each such pixel respectively corresponding to a possible left corner of one or more pallets, using a left corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

3. The method of claim 2, wherein the left corner template is defined by an M×

N window and said determining the first set of pixels comprises;

generating a left corner image having the first set of pixels by sliding the left corner template window concurrently over the set of horizontal cross correlation results and the set of vertical cross correlation results wherein;

during each placement of the left corner template window as it slides concurrently over the horizontal and vertical cross correlation results, receiving along one predefined vertical column of the window one or more corresponding horizontal cross correlation results, receiving along one predefined horizontal row of the window one or more corresponding vertical cross correlation results, identifying the minimum value of the cross correlation results in the one predefined vertical column and the one predefined horizontal row, and assigning the identified minimum value as a respective Score_{LowerLeftCorner} value in the first set of pixels.

4. The method of claim 3, wherein said one predefined vertical column comprises one outside vertical column and said one predefined horizontal row comprises one outside horizontal row.

5. The method of claim 4, wherein said assigning the identified minimum value comprises assigning the identified minimum value as the respective Score_{LowerLeftCorner} value for the pixel of the left corner image which is located at a position corresponding to an intersection of the one predefined vertical column and the one predefined horizontal row.

6. The method of claim 1, further comprising:

determining, using the computer, horizontal cross correlations between the gray scale image and a first step-edge template to generate a set of horizontal cross correlation results;

determining, using said computer, vertical cross correlations between the gray scale image and a second step-edge template to generate a set of vertical cross correlation results;

determining, using said computer, a first set of pixels, each such pixel respectively corresponding to a possible right corner of one or more pallets, using a right corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

7. The method of claim 6, wherein the right corner template is defined by an S×

T window and said determining the first set of pixels comprises;

generating a right corner image having the first set of pixels by sliding the right corner template window concurrently over the set of horizontal cross correlation results and the set of vertical cross correlation results wherein;

during each placement of the right corner template window as it slides concurrently over the horizontal and vertical cross correlation results, receiving along one predefined vertical column of the window one or more corresponding horizontal cross correlation results, receiving along one predefined horizontal row of the window one or more corresponding vertical cross correlation results, identifying a particular value of the cross correlation results in the one predefined vertical column and the one predefined horizontal row, and assigning the identified particular value as a respective Score_{LowerRightCorner} value in the first set of pixels.

8. The method of claim 7, wherein said assigning the identified particular value comprises assigning the identified particular value as the respective Score_{LowerRightCorner} value for the pixel of the right corner image which is located at a position corresponding to an intersection of the one predefined vertical column and the one predefined horizontal row.

9. The method of claim 7, wherein the respective identified particular value for one specific placement of the right corner template is a minimum value of the one or more corresponding vertical cross correlation results along the one predefined horizontal row and negated values of the one or more corresponding horizontal cross correlation results along the one predefined vertical column.

10. The method of claim 1, further comprising:

providing a Ro image comprising pixels that may generally correspond to an orthogonal distance from an origin point to one or more possible lines in the gray scale image;

providing a pixel location of the lower left corner associated with the possible pallet object;

defining, using said computer, a plurality of possible bottom pallet board lines passing through the lower left corner, each of the possible bottom pallet board lines being respectively oriented at one angle, φ

, within a range of angles to an axis of the Ro image; and

determining, using said computer, which of the plurality of possible bottom pallet board lines is most likely to be an actual bottom pallet board line passing through the lower left corner.

11. The method of claim 10, wherein said defining comprises calculating an orthogonal distance from the origin point in the Ro image to each of the possible bottom pallet board lines being respectively oriented at one angle from the following equation:

ρ

(φ

)=−

x_PalletCorner sin φ

+y_PalletCorner cos φ

wherein;

ρ

(φ

)=an orthogonal distance from the origin point in the Ro image to a possible bottom pallet board line at angle φ

;

φ

=one angle within the range of angles; and

(x_PalletCorner, y_PalletCorner)=coordinates in the Ro image for the lower left corner.

12. The method of claim 11, wherein said defining further comprises identifying respective points for each possible bottom pallet board line using ρ

(φ

) calculated for that bottom pallet board line.

13. The method of claim 12, further comprising evaluating the following expression for each point on each possible bottom pallet board line:

|ρ

_left(X,Y)−

ρ

(φ

)|≦

TOL2*ρ

(φ

)

wherein;

(X,Y)=one of the points on a possible bottom pallet board line;

ρ

_left(X,Y)=value from the Ro image at a point X,Y on the possible bottom pallet board line;

ρ

(φ

)=an orthogonal distance from the origin point in the Ro image to the possible bottom pallet board line at angle φ

;

φ

=the one angle within the range of angles; and

TOL2=an empirically determined value.

14. The method of claim 13, further comprising, for each possible bottom pallet line, incrementing a respective counter, Accumulator Count(φ

), each time the following expression is true for a point X, Y on the possible bottom pallet board line having the angle φ

;

|ρ

_left(X,Y)−

ρ

(φ

)|≦

TOL2*ρ

(φ

)

15. The method of claim 14, wherein the value Score_{BaseboardLine} is calculated according to:

16. The method according to claim 1, further comprising:

generating a first vertical projection of a first plurality of pixel values from the gray scale image, the first plurality of pixel values corresponding to a center stringer of the possible pallet object;

generating a second vertical projection of a second plurality of pixel values from the gray scale image, the second plurality of pixel values including a first set of pixel values corresponding with a first opening located to the left of the center stringer and a second set of pixel values corresponding to a second opening located to the right of the center stringer; and

wherein a value of Score_hole is generated based on the first and second vertical projections.

17. The method according to claim 16, wherein:

the first vertical projection includes only negative pixel values from the first plurality of pixel values;

the second vertical projection includes only negative pixel values from the first and second sets of pixel values.

18. The method of claim 17, further comprising:

providing a plurality of accumulators, Z_j, each accumulator corresponding to a respective column of the gray scale image wherein a value Z_j is calculated by summing only negative pixel values of the first plurality of pixel values occurring in that respective column of the gray scale image;

providing a plurality of accumulators, Y_i, each accumulator corresponding to a respective column of the gray scale image wherein a value Y_i is calculated by summing only negative pixel values of the second plurality of pixel values occurring in that respective column of the gray scale image; and

calculating Score_hole according to;

19. The method according to claim 1, wherein Score_Object is calculated according to:

20. A system for evaluating a possible pallet object in a gray scale image including one or more pallets, comprising:

a processor configured to execute instructions stored in an accessible memory;

a left corner scoring component configured to generate a first confidence score Score_{LowerLeftCorner} for a lower left corner associated with the possible pallet object;

a right corner scoring component configured to generate a second confidence score Score_{LowerRightCorner} for a lower right corner associated with the possible pallet object;

a line scoring component configured to generate a third confidence score Score_{BaseboardLine} for a bottom pallet board line associated with the possible pallet object;

a center stringer scoring component configured to generate a fourth confidence score Score_hole for a center stringer associated with the possible pallet object; and

a composite scoring component configured to calculate a composite score Score_Object based on the first, second, third and fourth confidence scores, wherein the composite score Score_Object represents a likelihood that the possible pallet object corresponds to an actual pallet structure in the gray scale image.

21. The system of claim 20, wherein the left corner scoring component is further configured to:

determine horizontal cross correlations between the gray scale image and a first step-edge template to generate a set of horizontal cross correlation results;

determine vertical cross correlations between the gray scale image and a second step-edge template to generate a set of vertical cross correlation results;

determine a first set of pixels, each such pixel respectively corresponding to a possible left corner of one or more pallets, using a left corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

22. The system of claim 21, wherein the left corner template is defined by an M×

N window and said left corner scoring component is further configured to;

generate a left corner image having the first set of pixels by sliding the left corner template window concurrently over the set of horizontal cross correlation results and the set of vertical cross correlation results wherein;

during each placement of the left corner template window as it slides concurrently over the horizontal and vertical cross correlation results, receiving along one predefined vertical column of the window one or more corresponding horizontal cross correlation results, receiving along one predefined horizontal row of the window one or more corresponding vertical cross correlation results, identifying the minimum value of the cross correlation results in the one predefined vertical column and the one predefined horizontal row, and assigning the identified minimum value as a respective Score_{LowerLeftCorner} value in the first set of pixels.

23. The system of claim 22, wherein said one predefined vertical column comprises one outside vertical column and said one predefined horizontal row comprises one outside horizontal row.

24. The system of claim 23, wherein the left corner scoring component is further configured to assign the identified minimum value as the respective Score_{LowerLeftCorner} value for the pixel of the left corner image which is located at a position corresponding to an intersection of the one predefined vertical column and the one predefined horizontal row.

25. The system of claim 20, wherein the right corner scoring component is further configured to:

determine horizontal cross correlations between the gray scale image and a first step-edge template to generate a set of horizontal cross correlation results;

determine vertical cross correlations between the gray scale image and a second step-edge template to generate a set of vertical cross correlation results;

determine a first set of pixels, each such pixel respectively corresponding to a possible right corner of one or more pallets, using a right corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

26. The system of claim 25, wherein the right corner template is defined by an S×

T window and said right corner scoring component is further configured to;

generate a right corner image having the first set of pixels by sliding the right corner template window concurrently over the set of horizontal cross correlation results and the set of vertical cross correlation results wherein;

during each placement of the right corner template window as it slides concurrently over the horizontal and vertical cross correlation results, receiving along one predefined vertical column of the window one or more corresponding horizontal cross correlation results, receiving along one predefined horizontal row of the window one or more corresponding vertical cross correlation results, identifying a particular value of the cross correlation results in the one predefined vertical column and the one predefined horizontal row, and assigning the identified particular value as a respective Score_{LowerRightCorner} value in the first set of pixels.

27. The system of claim 26, wherein said right corner scoring component is further configured to assign the identified particular value as the respective Score_{LowerRightCorner} value for the pixel of the right corner image which is located at a position corresponding to an intersection of the one predefined vertical column and the one predefined horizontal row.

28. The system of claim 26, wherein the respective identified particular value for one specific placement of the right corner template is a minimum value of the one or more corresponding vertical cross correlation results along the one predefined horizontal row and negated values of the one or more corresponding horizontal cross correlation results along the one predefined vertical column.

29. The system of claim 20, further comprising:

an image receiving component configured to receive a Ro image comprising pixels that may generally correspond to an orthogonal distance from an origin point to one or more possible lines in the gray scale image; and

wherein the line scoring component is further configured to;

identify a pixel location of the lower left corner associated with the possible pallet object;

define a plurality of possible bottom pallet board lines passing through the lower left corner, each of the possible bottom pallet board lines being respectively oriented at one angle, φ

, within a range of angles to an axis of the Ro image; and

determine which of the plurality of possible bottom pallet board lines is most likely to be an actual bottom pallet board line passing through the lower left corner.

30. The system of claim 29, wherein the line scoring component is further configured to calculate an orthogonal distance from the origin point in the Ro image to each of the possible bottom pallet board lines being respectively oriented at one angle from the following equation:

ρ

(φ

)=−

x_PalletCorner sin φ

+y_PalletCorner cos φ

wherein;

ρ

(φ

)=an orthogonal distance from the origin point in the Ro image to a possible bottom pallet board line at angle φ

;

φ

=one angle within the range of angles; and

(x_PalletCorner, y_PalletCorner)=coordinates in the Ro image for the lower left corner.

31. The system of claim 30, wherein the line scoring component is further configured to identify respective points for each possible bottom pallet board line using ρ

(φ

) calculated for that bottom pallet board line.

32. The system of claim 31, wherein the line scoring component is further configured to evaluate the following expression for each point on each possible bottom pallet board line:

|ρ

_left(X,Y)−

ρ

(φ

)|≦

TOL2*ρ

(φ

)

wherein;

(X,Y)=one of the points on a possible bottom pallet board line;

ρ

_left(X,Y)=value from the Ro image at a point X,Y on the possible bottom pallet board line;

ρ

(φ

)=an orthogonal distance from the origin point in the Ro image to the possible bottom pallet board line at angle φ

;

φ

=the one angle within the range of angles; and

TOL2=an empirically determined value.

33. The system of claim 32, wherein the line scoring component is further configured, for each possible bottom pallet line, to increment a respective counter, Accumulator Count(φ

), each time the following expression is true for a point X, Y on the possible bottom pallet board line having the angle φ

;

|ρ

_left(X,Y)−

ρ

(φ

)|≦

TOL2*ρ

(φ

)

34. The system of claim 33, wherein the line scoring component is configured to calculate Score_{BaseboardLine} according to:

35. The system according to claim 20, wherein the center stringer scoring component is further configured to:

generate a first vertical projection of a first plurality of pixel values from the gray scale image, the first plurality of pixel values corresponding to a center stringer of the possible pallet object;

generate a second vertical projection of a second plurality of pixel values from the gray scale image, the second plurality of pixel values including a first set of pixel values corresponding with a first opening located to the left of the center stringer and a second set of pixel values corresponding to a second opening located to the right of the center stringer; and

generate Score_hole based on the first and second vertical projections.

36. The system according to claim 35, wherein:

the first vertical projection includes only negative pixel values from the first plurality of pixel values;

the second vertical projection includes only negative pixel values from the first and second sets of pixel values.

37. The system of claim 36, wherein the center stringer scoring component is further configured to:

provide a plurality of accumulators, Z_j, each accumulator corresponding to a respective column of the gray scale image wherein a value Z_j is calculated by summing only negative pixel values of the first plurality of pixel values occurring in that respective column of the gray scale image;

provide a plurality of accumulators, Y_i, each accumulator corresponding to a respective column of the gray scale image wherein a value Y_i is calculated by summing only negative pixel values of the second plurality of pixel values occurring in that respective column of the gray scale image; and

Score_hole is calculated according to;

38. The system according to claim 20, wherein Score_Object is calculated according to:

39. A computer program product for evaluating a possible pallet object in a gray scale image including one or more pallets, comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to generate a first confidence score Score_{LowerLeftCorner} for a lower left corner associated with the possible pallet object;

computer readable program code configured to generate a second confidence score Score_{LowerRightCorner} for a lower right corner associated with the possible pallet object;

computer readable program code configured to generate a third confidence score Score_{BaseboardLine} for a bottom pallet board line associated with the possible pallet object;

computer readable program code configured to generate a fourth confidence score Score_hole for a center stringer associated with the possible pallet object; and

computer readable program code configured to calculate a composite score Score_Object based on the first, second, third and fourth confidence scores, wherein the composite score Score_Object represents a likelihood that the possible pallet object corresponds to an actual pallet structure in the gray scale image.

40. The computer program product of claim 39, further comprising:

computer readable program code configured to determine horizontal cross correlations between the gray scale image and a first step-edge template to generate a set of horizontal cross correlation results;

computer readable program code configured to determine vertical cross correlations between the gray scale image and a second step-edge template to generate a set of vertical cross correlation results; and

computer readable program code configured to determine a first set of pixels, each such pixel respectively corresponding to a possible left corner of one or more pallets, using a left corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

41. The computer program product of claim 40, wherein the left corner template is defined by an M×

N window and further comprising;

computer readable program code configured to generate a left corner image having the first set of pixels by sliding the left corner template window concurrently over the set of horizontal cross correlation results and the set of vertical cross correlation results wherein;

during each placement of the left corner template window as it slides concurrently over the horizontal and vertical cross correlation results, receiving along one predefined vertical column of the window one or more corresponding horizontal cross correlation results, receiving along one predefined horizontal row of the window one or more corresponding vertical cross correlation results, identifying the minimum value of the cross correlation results in the one predefined vertical column and the one predefined horizontal row, and assigning the identified minimum value as a respective Score_{LowerLeftCorner} value in the first set of pixels.

42. The computer program product of claim 41, wherein said one predefined vertical column comprises one outside vertical column and said one predefined horizontal row comprises one outside horizontal row.

43. The computer program product of claim 42, further comprising:

computer readable program code configured to assign the identified minimum value as the respective Score_{LowerLeftCorner} value for the pixel of the left corner image which is located at a position corresponding to an intersection of the one predefined vertical column and the one predefined horizontal row.

44. The computer program product of claim 39, further comprising:

computer readable program code configured to determine horizontal cross correlations between the gray scale image and a first step-edge template to generate a set of horizontal cross correlation results;

computer readable program code configured to determine vertical cross correlations between the gray scale image and a second step-edge template to generate a set of vertical cross correlation results; and

computer readable program code configured to determine a first set of pixels, each such pixel respectively corresponding to a possible right corner of one or more pallets, using a right corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

45. The computer program product of claim 44, wherein the right corner template is defined by an S×

T window further comprising;

computer readable program code configured to generate a right corner image having the first set of pixels by sliding the right corner template window concurrently over the set of horizontal cross correlation results and the set of vertical cross correlation results wherein;

during each placement of the right corner template window as it slides concurrently over the horizontal and vertical cross correlation results, receiving along one predefined vertical column of the window one or more corresponding horizontal cross correlation results, receiving along one predefined horizontal row of the window one or more corresponding vertical cross correlation results, identifying a particular value of the cross correlation results in the one predefined vertical column and the one predefined horizontal row, and assigning the identified particular value as a respective Score_{LowerRightCorner} value in the first set of pixels.

46. The computer program product of claim 45, further comprising:

computer readable program code configured to assign the identified particular value as the respective Score_{LowerRightCorner} value for the pixel of the right corner image which is located at a position corresponding to an intersection of the one predefined vertical column and the one predefined horizontal row.

47. The computer program product of claim 45, wherein the respective identified particular value for one specific placement of the right corner template is a minimum value of the one or more corresponding vertical cross correlation results along the one predefined horizontal row and negated values of the one or more corresponding horizontal cross correlation results along the one predefined vertical column.

48. The computer program product of claim 39, further comprising:

computer readable program code configured to receive a Ro image comprising pixels that may generally correspond to an orthogonal distance from an origin point to one or more possible lines in the gray scale image; and

computer readable program code configured to;

identify a pixel location of the lower left corner associated with the possible pallet object;

define a plurality of possible bottom pallet board lines passing through the lower left corner, each of the possible bottom pallet board lines being respectively oriented at one angle, φ

, within a range of angles to an axis of the Ro image; and

determine which of the plurality of possible bottom pallet board lines is most likely to be an actual bottom pallet board line passing through the lower left corner.

49. The computer program product of claim 48, further comprising:

computer readable program code configured to calculate an orthogonal distance from the origin point in the Ro image to each of the possible bottom pallet board lines being respectively oriented at one angle from the following equation;

ρ

(φ

)=−

x_PalletCorner sin φ

+y_PalletCorner cos φ

wherein;

ρ

(φ

)=an orthogonal distance from the origin point in the Ro image to a possible bottom pallet board line at angle φ

;

φ

=one angle within the range of angles; and

(x_PalletCorner, y_PalletCorner)=coordinates in the Ro image for the lower left corner.

50. The computer program product of claim 49, further comprising:

computer readable program code configured to identify respective points for each possible bottom pallet board line using ρ

(φ

) calculated for that bottom pallet board line.

51. The computer program product of claim 50, further comprising:

computer readable program code configured to evaluate the following expression for each point on each possible bottom pallet board line;

|ρ

_left(X,Y)−

ρ

(φ

)|≦

TOL2*ρ

(φ

)

wherein;

(X,Y)=one of the points on a possible bottom pallet board line;

ρ

_left(X,Y)=value from the Ro image at a point X,Y on the possible bottom pallet board line;

ρ

(φ

)=an orthogonal distance from the origin point in the Ro image to the possible bottom pallet board line at angle φ

;

φ

=the one angle within the range of angles; and

TOL2=an empirically determined value.

52. The computer program product of claim 51, further comprising:

computer readable program code configured, for each possible bottom pallet line, to increment a respective counter, Accumulator Count(φ

), each time the following expression is true for a point X, Y on the possible bottom pallet board line having the angle φ

;

|ρ

_left(X,Y)−

ρ

(φ

)|≦

TOL2*ρ

(φ

)

53. The computer program product of claim 52, further comprising:

computer readable program code configured to calculate Score_{BaseboardLine} according to;

54. The computer program product of claim 39, further comprising:

computer readable program code configured to;

generate a first vertical projection of a first plurality of pixel values from the gray scale image, the first plurality of pixel values corresponding to a center stringer of the possible pallet object;

generate a second vertical projection of a second plurality of pixel values from the gray scale image, the second plurality of pixel values including a first set of pixel values corresponding with a first opening located to the left of the center stringer and a second set of pixel values corresponding to a second opening located to the right of the center stringer; and

generate Score_hole based on the first and second vertical projections.

55. The computer program product according to claim 54, wherein:

the first vertical projection includes only negative pixel values from the first plurality of pixel values;

the second vertical projection includes only negative pixel values from the first and second sets of pixel values.

56. The computer program product of claim 55, further comprising:

computer readable program code configured to;

provide a plurality of accumulators, Z_j, each accumulator corresponding to a respective column of the gray scale image wherein a value Z_j is calculated by summing only negative pixel values of the first plurality of pixel values occurring in that respective column of the gray scale image;

provide a plurality of accumulators, Y_i, each accumulator corresponding to a respective column of the gray scale image wherein a value Y_i is calculated by summing only negative pixel values of the second plurality of pixel values occurring in that respective column of the gray scale image; and

and calculate Score_hole according to;

57. The computer program product of claim 39, wherein Score_Object is calculated according to: