Controlling truck forks based on identifying and tracking multiple objects in an image scene

• US 9,025,827 B2
• Filed: 10/16/2012
• Issued: 05/05/2015
• Est. Priority Date: 10/19/2011
• Status: Active Grant

• Current Assignee: Crown Equipment Corporation
• Original Assignee: Crown Equipment Corporation
• Inventors: Holeva, Lee F., Elston, Edwin R., Jensen, Eric L., Seelinger, Michael J., Yoder, John-David S.
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/652,622
• Publication Number:
  US 20130101173A1
• Time in Patent Office: 931 Days
• Field of Search: None
• US Class Current: 382/104
• CPC Class Codes: B66F 9/122 longitudinally movable
• CPC Class Codes: B66F 9/24 Electrical devices or systems
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A method is provided for controlling forks of a vehicle. The method may comprise: while the forks of the vehicle are moving vertically, acquiring a series of images of a scene of a physical environment in which a plurality of pallets are visible; identifying in each image, by a computer system, one or more scored candidate objects, each potentially corresponding to a respective one of the plurality of pallets; for each of the one or more scored candidate objects, tracking, by the computer system, a respective location in each of at least two images of the series; determining for each of the one or more scored candidate objects a respective, associated height in the physical environment; and stopping, by the computer system, the forks of the vehicle at a height in the physical environment based on the height of a specific one of the scored candidate objects.

Claims

1. A method for controlling forks of a vehicle, comprising:

while the forks of the vehicle are moving vertically, acquiring a series of images of a scene of a physical environment in which a plurality of pallets are visible;

identifying in each image, by a computer system, one or more scored candidate objects, each potentially corresponding to a respective one of the plurality of pallets, wherein each scored candidate object has a respective associated score that is a confidence value that the scored candidate object corresponds to one or more physical structures of a pallet;

for each of the one or more scored candidate objects, tracking, by the computer system, a respective location in each of at least two images of the series;

determining for each of the one or more scored candidate objects a respective, associated height in the physical environment, wherein the respective, associated height for each particular one of the scored candidate objects is based on the respective locations in the each of the at least two images for that particular one scored candidate object; and

stopping, by the computer system, the forks of the vehicle at a height in the physical environment based on the height of a specific one of the scored candidate objects.

2. The method of claim 1, wherein the series of images is acquired using a camera that is moving concurrently with the forks such that each image in the series is acquired at a different, corresponding camera position.

3. The method of claim 1, wherein acquiring the series of images, comprises:

capturing, by a single imaging camera, a series of gray scale images of the scene.

4. The method of claim 1, wherein the plurality of pallets are arranged as a single column of vertically spaced pallets.

5. The method of claim 1, wherein identifying in each image each of the one or more respective scored candidate objects, comprises:

identifying a plurality of features in each image, wherein each identified feature potentially corresponds to a respective physical structure of a pallet.

6. The method of claim 5, wherein the respective physical structures of a pallet comprise a center vertical stringer, a baseboard line, and a pair of fork openings.

7. The method of claim 6, comprising:

assigning a first confidence score to the identified feature corresponding to the center vertical stringer;

assigning a second confidence score to the identified feature corresponding to the baseboard line;

assigning a third confidence score to the identified feature corresponding to the pair of fork opening; and

identifying the plurality of features as a scored candidate object in a particular image based on a composite score calculated from the first, second, and third confidence scores, wherein the composite score comprises the respective associated score that is a confidence value that the scored candidate object corresponds to one or more physical structures of a pallet.

8. The method of claim 1, comprising:

maintaining a list having a respective record for each scored candidate object, wherein the respective record for a particular scored candidate object comprises the respective location in the at least two images for that particular scored candidate object.

9. The method of claim 8, wherein tracking the respective location in the at least two images for each of the one or more scored candidate objects comprises:

in response to a most-recent image in the series of images being acquired, analyzing the two most recently acquired images in the series of images to determine if a particular one of the one or more scored candidate objects in the second-most-recent image has a matching scored candidate object in the most-recent image; and

adding an entry in the particular one scored candidate object'"'"'s respective record based on the respective location of the matching scored candidate object in the most-recent image.

10. The method of claim 9, wherein analyzing the two most recently acquired images comprises:

identifying in the most-recent image one or more possible scored candidate objects;

determining a respective measured location for each possible scored candidate object in the most-recent image; and

for the particular one of the one or more scored candidate objects;

calculating a respective predicted location in the most-recent image, wherein the respective predicted location is estimated based on the respective location of the particular one scored candidate object in the second-most-recent image and movement of the camera between acquisition of the second-most-recent image and the most-recent image,

calculating a respective prediction window, wherein the respective prediction window defines a set of pixel locations in the most-recent image nearby the respective predicted location, and

for any possible scored candidate object in the most-recent image having a respective measured location within the respective prediction window, tentatively matching that possible scored candidate object in the most-recent image with the particular one scored candidate object in the second-most-recent image.

11. The method of claim 10, comprising:

when a plurality of the one or more possible scored candidate objects tentatively match the particular one scored candidate object, selecting one possible scored candidate object from the plurality to definitively match the particular one scored candidate object, or

when only one of the one or more possible scored candidate objects tentatively matches the particular one scored candidate object, selecting the tentatively matched possible scored candidate object to definitively match the particular one scored candidate object.

12. The method of claim 11, wherein calculating the respective predicted location for the particular one scored candidate object comprises using a Kalman filter.

13. The method of claim 12, wherein a Kalman gain for the Kalman filter is based on a difference between the respective predicted location and the respective measured location for the definitively matched possible scored candidate object.

14. The method of claim 2, wherein determining the respective, associated height in the physical environment based on the respective locations for that particular one scored candidate object comprises:

identifying a world coordinate origin location in the physical environment;

identifying for each of the at least two images a corresponding position of the camera in the physical environment relative to the world coordinate origin; and

determining the respective, associated height relative to the world coordinate origin based on;

a) the respective locations from each of the at least two images for that particular one scored candidate object, and

b) the corresponding position of the camera for each of the at least two images.

15. The method of claim 14, wherein determining the respective, associated height relative to the world coordinate origin, comprises:

calculating an initial respective, associated height using a first subset of the respective locations for that particular one scored candidate object, wherein a second subset of locations comprises all of the respective locations for that particular one score candidate object other than those locations in the first subset;

calculating an error measurement for the initial respective, associated height;

using the initial respective, associated height, calculating a respective error measurement associated with each of the locations of the second subset; and

identifying a third subset of locations comprising those locations of the second subset having an associated, respective error measurement less than the error measurement.

16. The method of claim 14, wherein the camera is coupled to a portion of the vehicle.

17. The method of claim 16, wherein identifying for each of the at least two images the corresponding position of the camera in the physical environment relative to the world coordinate origin comprises:

compensating for movement of the vehicle that occurs between acquisition of each image.

18. The method of claim 1, wherein the computer system comprises an image analysis computer and a vehicle computer, the method further comprising:

communicating the height of the specific one of the scored candidate objects from the image analysis computer to the vehicle computer by communicating a respective, associated height of the specific one of the scored candidate objects and a predetermined offset value.

19. The method of claim 1, wherein the computer system comprises an image analysis computer and a vehicle computer, the method further comprising:

communicating the height of the specific one of the scored candidate objects from the image analysis computer to the vehicle computer, wherein communicating comprises;

identifying a current height of the forks;

of the one or more scored candidate objects, determining a particular scored candidate object with its respective, associated height closest to, and greater than, the current height of the forks, the particular scored candidate object defining the specific one scored candidate object; and

communicating the respective, associated height of that particular scored candidate object from the image analysis computer to the vehicle computer.

20. The method of claim 19, wherein communicating the height of the specific one of the scored candidate objects from the image analysis computer to the vehicle computer, comprises:

updating which of the respective, associated heights is the communicated respective, associated height based on a current height of the forks.

21. The method of claim 1, wherein stopping the forks of the vehicle at a height in the physical environment comprises:

receiving input from a user-operated control device of the vehicle;

monitoring a height of the forks as the forks are moving vertically; and

stopping vertical movement of the forks when the height of the forks substantially equals a communicated height of the specific one of the scored candidate objects.

22. A system for controlling forks of a vehicle, comprising:

an imaging camera configured to acquire, while the forks of the vehicle are moving vertically, a series of images of a scene of a physical environment in which a plurality of pallets are visible;

a computer configured to execute instructions in an accessible memory;

the computer when executing instructions stored in the accessible memory;

identifies in each image one or more scored candidate objects, each potentially corresponding to a respective one of the plurality of pallets, wherein each scored candidate object has a respective associated score that is a confidence value that the scored candidate object corresponds to one or more physical structures of a pallet;

tracks a respective location in each of at least two images of the series for each of the one or more scored candidate objects, and

determines for each of the one or more scored candidate objects a respective, associated height in the physical environment, wherein the respective, associated height for each particular one of the scored candidate objects is based on the respective locations in the each of the at least two images for that particular one scored candidate object; and

a fork controller configured to stop the forks of the vehicle at a height in the physical environment based on the height of a specific one of the scored candidate objects.

23. The system of claim 22, wherein the imaging camera comprises a camera configured to move concurrently with the forks such that each image in the series is acquired at a different, corresponding camera position.

24. The system of claim 22, wherein the series of images comprises a series of gray scale images of the scene captured by a single camera.

25. The system of claim 22, wherein the plurality of pallets are arranged as a single column of vertically spaced pallets.

26. The system of claim 22, wherein the computer when executing instructions stored in the accessible memory identifies a plurality of features in each image, wherein each identified feature potentially corresponds to a respective physical structure of a pallet.

27. The system of claim 26, wherein the respective physical structures of a pallet comprise a center vertical stringer, a baseboard line, and a pair of fork openings.

28. The system of claim 27, wherein the computer when executing instructions stored in the accessible memory identifies each of the one or more scored candidate object according to:

assigning a first confidence score to the identified feature corresponding to the center vertical stringer;

assigning a second confidence score to the identified feature corresponding to the baseboard line;

assigning a third confidence score to the identified feature corresponding to the pair of fork opening; and

identifying the plurality of features as a scored candidate object in a particular image based on a composite score calculated from the first, second, and third confidence scores, wherein the composite score comprises the respective associated score that is a confidence value that the scored candidate object corresponds to one or more physical structures of a pallet.

29. The system of claim 22, comprising:

a memory configured to store a list having a respective record for each scored candidate object, wherein the respective record for a particular scored candidate object comprises the respective location in the at least two images for that particular scored candidate object.

30. The system of claim 29, wherein the computer when executing instructions stored in the accessible memory:

in response to a most-recent image in the series of images being acquired, analyzes the two most recently acquired images in the series of images to determine if a particular one of the one or more scored candidate objects in the second-most-recent image has a matching scored candidate object in the most-recent image; and

adds an entry in the particular one scored candidate object'"'"'s respective record based on the respective location of the matching scored candidate object in the most-recent image.

31. The system of claim 30, wherein the computer when executing instructions stored in the accessible memory:

identifies in the most-recent image one or more possible scored candidate objects;

determines a respective measured location for each possible scored candidate object in the most-recent image; and

for the particular one of the one or more scored candidate objects;

calculates a respective predicted location in the most-recent image, wherein the respective predicted location is estimated based on the respective location of the particular one scored candidate object in the second-most-recent image and movement of the camera between acquisition of the second-most-recent image and the most-recent image,

calculates a respective prediction window, wherein the respective prediction window defines a set of pixel locations in the most-recent image nearby the respective predicted location, and

for any possible scored candidate object in the most-recent image having a respective measured location within the respective prediction window, tentatively matches that possible scored candidate object in the most-recent image with the particular one scored candidate object in the second-most-recent image.

32. The system of claim 31, wherein the computer when executing instructions stored in the accessible memory:

when a plurality of the one or more possible scored candidate objects tentatively match the particular one scored candidate object, selects one possible scored candidate object from the plurality to definitively match the particular one scored candidate object, or

when only one of the one or more possible scored candidate objects tentatively matches the particular one scored candidate object, selects the tentatively matched possible scored candidate object to definitively match the particular one scored candidate object.

33. The system of claim 32, wherein the computer when executing instructions stored in the accessible memory calculates the respective predicted location for the particular one scored candidate object using a Kalman filter.

34. The system of claim 33, wherein a Kalman gain for the Kalman filter is based on a difference between the respective predicted location and the respective measured location for the definitively matched possible scored candidate object.

35. The system of claim 23, wherein the computer when executing instructions stored in the accessible memory:

identifies a world coordinate origin location in the physical environment;

identifies for each of the at least two images a corresponding position of the camera in the physical environment relative to the world coordinate origin; and

determines the respective, associated height relative to the world coordinate origin based on;

a) the respective locations from each of the at least two images for that particular one scored candidate object, and

b) the corresponding position of the camera for each of the at least two images.

36. The system of claim 35, wherein the computer when executing instructions stored in the accessible memory:

calculates an initial respective, associated height using a first subset of the respective locations for that particular one scored candidate object, wherein a second subset of locations comprises all of the respective locations for that particular one score candidate object other than those locations in the first subset;

calculates an error measurement for the initial respective, associated height;

using the initial respective, associated height, calculates a respective error measurement associated with each of the locations of the second subset; and

identifies a third subset of locations comprising those locations of the second subset having an associated, respective error measurement less than the error measurement.

37. The system of claim 35, wherein the camera is coupled to a portion of the vehicle.

38. The system of claim 37, wherein the computer when executing instructions stored in the accessible memory compensates for movement of the vehicle that occurs between acquisition of each the at least two images.

39. The system of claim 22, further comprising:

a communication link configured to provide the height of the specific one of the scored candidate objects from the computer to the fork controller by communicating a respective, associated height of the specific one of the scored candidate objects and a predetermined offset value.

40. The system of claim 22, wherein the computer when executing instructions stored in the accessible memory provides the height of the specific one of the scored candidate objects to the fork controller by:

identifying a current height of the forks;

of the one or more scored candidate objects, determining a particular scored candidate object with its respective, associated height closest to, and greater than, the current height of the forks, the particular scored candidate object defining the specific one scored candidate object; and

communicating the respective, associated height of that particular scored candidate object to the fork controller.

41. The system of claim 40, wherein the computer when executing instructions stored in the accessible memory updates which of the respective, associated heights is the communicated respective, associated height based on a current height of the forks.

42. The system of claim 22, wherein the fork controller is further configured to:

receive input from a user-operated control device of the vehicle;

monitor a height of the forks as the forks are moving vertically; and

stop vertical movement of the forks when the height of the forks substantially equals a communicated height of the specific one of the scored candidate objects.

43. A computer program product for controlling forks of a vehicle, comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to, while the forks of the vehicle are moving vertically, acquire a series of images of a scene of a physical environment in which a plurality of pallets are visible;

computer readable program code configured to identify in each image one or more scored candidate objects, each potentially corresponding to a respective one of the plurality of pallets, wherein each scored candidate object has a respective associated score that is a confidence value that the scored candidate object corresponds to one or more physical structures of a pallet;

computer readable program code configured to, for each of the one or more scored candidate objects, track a respective location in each of at least two images of the series;

computer readable program code configured to determine for each of the one or more scored candidate objects a respective, associated height in the physical environment, wherein the respective, associated height for each particular one of the scored candidate objects is based on the respective locations in the each of the at least two images for that particular one scored candidate object; and

computer readable program code configured to stop the forks of the vehicle at a height in the physical environment based on the height of a specific one of the scored candidate objects.

44. The computer program product of claim 43, further comprising:

computer readable program code configured to acquire the series of images using a camera that is moving concurrently with the forks such that each image in the series is acquired at a different, corresponding camera position.

45. The computer program product of claim 43, wherein the series of images comprises a series of gray scale images of the scene captured by a single camera.

46. The computer program product of claim 43, wherein the plurality of pallets are arranged as a single column of vertically spaced pallets.

47. The computer program product of claim 43, further comprising:

computer readable program code configured to identify a plurality of features in each image, wherein each identified feature potentially corresponds to a respective physical structure of a pallet.

48. The computer program product of claim 47, wherein the respective physical structures of a pallet comprise a center vertical stringer, a baseboard line, and a pair of fork openings.

49. The computer program product of claim 48, further comprising:

computer readable program code configured to;

assign a first confidence score to the identified feature corresponding to the center vertical stringer;

assign a second confidence score to the identified feature corresponding to the baseboard line;

assign a third confidence score to the identified feature corresponding to the pair of fork opening; and

identify the plurality of features as a scored candidate object in a particular image based on a composite score calculated from the first, second, and third confidence scores, wherein the composite score comprises the respective associated score that is a confidence value that the scored candidate object corresponds to one or more physical structures of a pallet.

50. The computer program product of claim 43, further comprising:

computer readable program code configured to maintain a list having a respective record for each scored candidate object, wherein the respective record for a particular scored candidate object comprises the respective location in the at least two images for that particular scored candidate object.

51. The computer program product of claim 50, further comprising:

computer readable program code configured to, in response to a most-recent image in the series of images being acquired, analyze the two most recently acquired images in the series of images to determine if a particular one of the one or more scored candidate objects in the second-most-recent image has a matching scored candidate object in the most-recent image; and

computer readable program code configured to add an entry in the particular one scored candidate object'"'"'s respective record based on the respective location of the matching scored candidate object in the most-recent image.

52. The computer program product of claim 51, further comprising:

computer readable program code configured to;

identify in the most-recent image one or more possible scored candidate objects;

determine a respective measured location for each possible scored candidate object in the most-recent image; and

for the particular one of the one or more scored candidate objects;

calculate a respective predicted location in the most-recent image, wherein the respective predicted location is estimated based on the respective location of the particular one scored candidate object in the second-most-recent image and movement of the camera between acquisition of the second-most-recent image and the most-recent image,

calculate a respective prediction window, wherein the respective prediction window defines a set of pixel locations in the most-recent image nearby the respective predicted location, and

for any possible scored candidate object in the most-recent image having a respective measured location within the respective prediction window, tentatively match that possible scored candidate object in the most-recent image with the particular one scored candidate object in the second-most-recent image.

53. The computer program product of claim 52, further comprising computer readable program code configured to:

when a plurality of the one or more possible scored candidate objects tentatively match the particular one scored candidate object, select one possible scored candidate object from the plurality to definitively match the particular one scored candidate object, or

when only one of the one or more possible scored candidate objects tentatively matches the particular one scored candidate object, select the tentatively matched possible scored candidate object to definitively match the particular one scored candidate object.

54. The computer program product of claim 53, further comprising:

computer readable program code configured to implement a Kalman filter to calculate the respective predicted location for the particular one scored candidate object.

55. The computer program product of claim 54, wherein a Kalman gain for the Kalman filter is based on a difference between the respective predicted location and the respective measured location for the definitively matched possible scored candidate object.

56. The computer program product of claim 44, further comprising:

computer readable program code configured to identify a world coordinate origin location in the physical environment;

computer readable program code configured to identify for each of the at least two images a corresponding position of the camera in the physical environment relative to the world coordinate origin; and

computer readable program code configured to determine the respective, associated height relative to the world coordinate origin based on;

a) the respective locations from each of the at least two images for that particular one scored candidate object, and

b) the corresponding position of the camera for each of the at least two images.

57. The computer program product of claim 56, further comprising:

computer readable program code configured to calculate an initial respective, associated height using a first subset of the respective locations for that particular one scored candidate object, wherein a second subset of locations comprises all of the respective locations for that particular one score candidate object other than those locations in the first subset;

computer readable program code configured to calculate an error measurement for the initial respective, associated height;

computer readable program code configured to, using the initial respective, associated height, calculate a respective error measurement associated with each of the locations of the second subset; and

computer readable program code configured to identify a third subset of locations comprising those locations of the second subset having an associated, respective error measurement less than the error measurement.

58. The computer program product of claim 56, wherein the camera is coupled to a portion of the vehicle.

59. The computer program product of claim 58, further comprising:

computer readable program code configured to compensate for movement of the vehicle that occurs between acquisition of the at least two images.

60. The computer program product of claim 43, further comprising:

computer readable program code configured to provide the height of the specific one of the scored candidate objects from the image analysis component to the fork controller by communicating a respective, associated height of the specific one of the scored candidate objects and a predetermined offset value.

61. The computer program product of claim 43, further comprising:

computer readable program code configured to provide the height of the specific one of the scored candidate objects to the fork controller, by;

identifying a current height of the forks;

of the one or more scored candidate objects, determining a particular scored candidate object with its respective, associated height closest to, and greater than, the current height of the forks, the particular scored candidate object defining the specific one scored candidate object; and

communicating the respective, associated height of that particular scored candidate object to the fork controller.

62. The computer program product of claim 61, further comprising:

computer readable program code configured to update which of the respective, associated heights is the communicated respective, associated height based on a current height of the forks.

63. The computer program product of claim 43, further comprising:

computer readable program code configured to receive input from a user-operated control device of the vehicle;

computer readable program code configured to monitor a height of the forks as the forks are moving vertically; and

computer readable program code configured to stop vertical movement of the forks when the height of the forks substantially equals a communicated height of the specific one of the scored candidate objects.