System and method of material handling using one or more imaging devices on the transferring vehicle to control the material distribution into the storage portion of the receiving vehicle

• US 9,457,971 B2
• Filed: 02/11/2013
• Issued: 10/04/2016
• Est. Priority Date: 02/10/2012
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, Deere & Company
• Original Assignee: Carnegie Mellon University, Deere & Company
• Inventors: Bonefas, Zachary T., Chaney, Mark M., Johnson, Jeremiah K., Bruns, Aaron J., Herman, Herman, Vallespi-Gonzalez, Carlos, Mora, Jose Gonzalez
• Primary Examiner(s): BEAUCHAINE, MARK J
• Application Number:
  US14/377,383
• Publication Number:
  US 20150023775A1
• Time in Patent Office: 1,331 Days
• Field of Search: 700/214, 700/275, 700/302, 414/334, 414/335, 414/345, 414/373, 414/374, 414/376, 414/389, 414/390, 414/397, 414/813
• US Class Current: 1/1
• CPC Class Codes: A01D 43/087 with controllable discharge spout
• CPC Class Codes: B60P 1/42 mounted on the load-transporting element
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B62D 6/001 the torque NOT being among the input parameters
• CPC Class Codes: B65B 1/48 Checking volume of filled material
• CPC Class Codes: B65G 67/22 Loading moving vehicles
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0293 Convoy travelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

First imaging device collects first image data, whereas second imaging device collects second image data of a storage portion. A container identification module identifies a container perimeter of the storage portion in at least one of the collected first image data and the collected second image data. A spout identification module is adapted to identify a spout of the transferring vehicle in the collected image data. An image data evaluator determines whether to use the first image data, the second image data, or both based on an evaluation of the intensity of pixel data or ambient light conditions. An alignment module is adapted to determine the relative position of the spout and the container perimeter and to generate command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central zone or a target zone of the container perimeter.

Claims

1. A method for facilitating the transfer of material from a transferring vehicle having a material distribution end to a receiving vehicle having a bin to store the transferred material, the method comprising the steps of:

a. obtaining image data from at least one imaging device facing the bin, wherein the at least one imaging device is mounted on the transferring vehicle;

b. identifying and locating the bin from the image data using an image processing module;

c. identifying a spout from the image data using the image processing module to determining a location and orientation of the spout;

d. detecting a representation of the fill level or volumetric distribution of the material in the bin;

e. determining a position of the spout relative to the bin using an alignment module based on an identified location of the bin and the identified location and orientation of the spout;

f. aligning the spout over a current target area of the bin requiring the material by generating command data used by at least one of a vehicle controller and a spout controller,

wherein the command data is based in part on the position of the spout relative to the bin;

g. determining subsequent target areas of the bin that require material based on the representation of the fill level or volumetric distribution of the material in the bin;

h. transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle;

i. detecting when the current target area of the bin is filled with the material;

j. repeating steps f-i until the subsequent target areas of the bin are filled; and

k. terminating the transfer of the material from the transferring vehicle to the receiving vehicle.

2. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is one-dimensional.

3. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is two-dimensional.

4. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is three-dimensional.

5. The method according to claim 1, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the bin further comprises the steps of:

receiving data from a distributed fill state sensor; and

generating the representation of the fill level or volumetric distribution of the material in the bin based on the distributed fill state sensor.

6. The method according to claim 1, wherein the material is agricultural material.

7. The method according to claim 1, wherein the material is a mineral material.

8. The method according to claim 1, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the bin further comprises the steps of:

receiving rectified image data from the at least one imaging device;

generating disparity image data based on the rectified image data; and

generating the representation of the fill level or volumetric distribution of the material in the bin using range data based on the disparity image data.

9. The method according to claim 8, wherein the step of receiving rectified image data from one or more imaging devices further comprises receiving a rectified image data from an imaging device on the transferring vehicle, wherein there are no imaging devices on the receiving vehicle.

10. The method according to claim 8, wherein the step of receiving rectified image data from the at least one imaging device further comprises:

receiving a first rectified image data from a first imaging device on the transferring vehicle; and

receiving a second rectified image data from a second imaging device on the transferring vehicle,

wherein there are no imaging devices on the receiving vehicle.

11. The method according to claim 1, wherein the step of determining subsequent target areas of the bin that require material based on the representation of the fill level or volumetric distribution of the material in the bin comprises the steps of:

developing a target zone matrix of the bin, wherein each target zone of the matrix is identified by a pre-established set of coordinates related to the bin;

identifying target zones of the matrix that are filled and not filled based on the representation of the fill level or volumetric distribution of the material in the bin; and

determining the pre-established set of coordinates of a subsequent target zone to be filled and the pre-established set of coordinates of the current target zone over which the material distribution end is positioned.

12. The method according to claim 11, wherein the step of aligning the spout over a current target area of the bin requiring the material comprises the step of actuating a control mechanism to move the spout over the pre-established set of coordinates of a subsequent target zone to be filled.

13. The method according to claim 12, wherein the step of actuating a control mechanism to move the spout over the pre-established set of coordinates of a subsequent target zone to be filled comprises the steps of:

establishing a current lateral spout angle of the spout;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current lateral spout angle to form a new lateral spout angle;

transmitting the new lateral spout angle to a spout controller of the transferring vehicle; and

actuating the spout controller from the current lateral spout angle to the new lateral spout angle,

whereby the spout is aligned over the subsequent target zone to be filled.

14. The method according to claim 11, wherein the step of aligning the spout over a current target area of the bin requiring the material comprises the step of changing a lateral offset between the receiving vehicle and the transferring vehicle to move the spout from over the current target zone to over the subsequent target zone to be filled.

15. The method according to claim 14, wherein the step of changing a lateral offset between the receiving vehicle and the transferring vehicle comprises the step of:

establishing a current lateral offset between the receiving vehicle and the transferring vehicle;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current lateral offset to form a new lateral offset;

transmitting the new lateral offset to a steering controller of the transferring vehicle; and

steering the transferring vehicle to create the new lateral offset between the receiving vehicle and the transferring vehicle,

whereby the spout is aligned the over the subsequent target zone to be filled.

16. The method according to claim 11, wherein the step of aligning the spout over a current target area of the bin requiring the material comprises the step of changing a fore/aft offset between the receiving vehicle and the transferring vehicle to move the spout from over the current target zone to over the subsequent target zone to be filled.

17. The method according to claim 16, wherein the step of changing a fore/aft offset between the receiving vehicle and the transferring vehicle comprises the step of:

establishing an initial fore/aft offset between the receiving vehicle and the transferring vehicle;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current fore/aft offset to form a new fore/aft offset;

transmitting the new fore/aft offset to a propulsion controller of the transferring vehicle, or a braking controller of the transferring vehicle; and

accelerating or braking the transferring vehicle to create the new fore/aft offset between the receiving vehicle and the transferring vehicle,

whereby the spout is aligned the over the subsequent target zone to be filled.

18. The method according to claim 1, wherein the step of transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle comprises the step of actuating an auger.

19. The method according to claim 18, wherein the step of terminating the transfer of the material from the transferring vehicle to the receiving vehicle comprises the step of stopping the auger.

20. The method according to claim 1, wherein the step of transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle comprises the step of actuating the spout.

21. The method according to claim 20, wherein the step of terminating the transfer of the material from the transferring vehicle to the receiving vehicle comprises the step of stopping the spout.

22. The method according to claim 1, wherein the step of detecting when the current target area of the bin is filled with the material comprises the step of detecting a representation of the fill level or volumetric distribution of the material in the current target zone of the bin.

23. The method according to claim 22, wherein the step of detecting a representation of the fill level or volumetric distribution of the material in the current target zone of the bin further comprises the steps of:

receiving data from a distributed fill state sensor; and

generating the representation of the fill level or volumetric distribution of the material in the current target zone of the bin based on the distributed fill state sensor data.

24. The method according to claim 22, wherein the step of detecting a representation of the fill level or volumetric distribution of the material in the current target zone of the bin further comprises the steps of:

receiving rectified image data from the at least one imaging device;

generating disparity image data based on the rectified image data; and

generating the representation of the fill level or volumetric distribution of the material in the bin using range data based on the disparity image data.

25. The method according to claim 24, wherein the step of receiving rectified image data from the at least one imaging device further comprises receiving a rectified image data from an imaging device on the transferring vehicle, wherein there are no imaging devices on the receiving vehicle.

26. The method according to claim 24, wherein the step of receiving rectified image data from the at least one imaging device further comprises:

receiving a first rectified image data from a first imaging device on the transferring vehicle; and

receiving a second rectified image data from a second imaging device on the transferring vehicle,

wherein there are no imaging devices on the receiving vehicle.