System and method of material handling using one imaging device on the receiving vehicle to control the material distribution into the storage portion of the receiving vehicle

• US 9,415,953 B2
• Filed: 08/07/2014
• Issued: 08/16/2016
• Est. Priority Date: 02/10/2012
• Status: Active Grant

• Current Assignee: Deere & Company
• Original Assignee: Deere & Company
• Inventors: Bonefas, Zachary T.
• Primary Examiner(s): Butler, Rodney
• Assistant Examiner(s): BRUSHABER, FREDERICK M
• Application Number:
  US14/454,175
• Publication Number:
  US 20140350801A1
• Time in Patent Office: 740 Days
• Field of Search: 701/50, 701/23, 701/301, 340/436, 340/435, 348/89
• US Class Current: 1/1
• CPC Class Codes: A01D 43/087 with controllable discharge spout
• CPC Class Codes: B60P 1/42 mounted on the load-transporting element
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B62D 6/001 the torque NOT being among the input parameters
• CPC Class Codes: B65B 1/48 Checking volume of filled material
• CPC Class Codes: B65G 67/22 Loading moving vehicles
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0293 Convoy travelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A single imaging device collects image data of a storage portion. A container module identifies a container perimeter of the storage portion in at least one of the collected image data. A spout module is adapted to identify a spout of the transferring vehicle in the collected image data. An arbiter determines whether to use the image data based on an evaluation of the intensity of pixel data or ambient light conditions. An alignment module is adapted to determine the relative position of the spout and the container perimeter and to generate command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central or target zone of the container perimeter.

Claims

1. A method for facilitating the transfer of material from a transferring vehicle having a material distribution end to a receiving vehicle having a bin to store transferred material, the method comprising the steps of:

a. capturing image data representing a field of view containing a material distribution end and a bin with a single imaging device;

b. identifying and locating the bin,

wherein the bin is located using the image data by identifying a linear set of pixels associated with an edge of the bin,

wherein the material distribution end is located by identifying pixels associated with the material distribution end in the image data,

wherein distances from the imaging device to the linear set of pixels and the pixels associated with the material distribution end are estimated using the image data, thereby providing a location of the bin relative to the material distribution end;

c. detecting a representation of the fill level or volumetric distribution of the material in the bin;

d. sending the representation and the location of the bin to an alignment module;

e. generating commands in the alignment module capable of being used by at least one of the receiving vehicle and the transferring vehicle to align the material distribution end over a current target area of the bin requiring the material;

f. determining subsequent target areas of the bin that require material based on the representation of the fill level or volumetric distribution of the material in the bin and a desired fill pattern to fill the bin;

g. transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle;

h. detecting when the current target area of the bin is filled with the material; and

i. repeating steps e-h until the subsequent target areas of the bin are filled per the desired fill pattern.

2. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is one-dimensional.

3. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is two-dimensional.

4. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is three-dimensional.

5. The method according to claim 1, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the bin further comprises the steps of:

receiving data from a distributed fill state sensor;

generating the representation of the fill level or volumetric distribution of the material in the bin based on the distributed fill state sensor.

6. The method according to claim 1, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the bin further comprises the steps of:

receiving rectified image data from the single imaging device;

generating disparity image data based on the rectified image data; and

generating the representation of the fill level or volumetric distribution of the material in the bin using range data based on the disparity image data.

7. The method according to claim 6, wherein the step of receiving rectified image data from the single imaging device further comprises receiving a rectified image data from an imaging device on the receiving vehicle, wherein there are no imaging devices on the transferring vehicle.

8. The method according to claim 1, wherein the step of determining subsequent target areas of the bin that require material based on the representation of the fill level or volumetric distribution of the material in the bin and a desired fill pattern to fill the bin comprises the steps of:

developing a target zone matrix of the bin, wherein each target zone of the matrix is identified by a pre-established set of coordinates related to the bin;

identifying target zones of the matrix that are filled and not filled based on the representation of the fill level or volumetric distribution of the material in the bin; and

determining the pre-established set of coordinates of a subsequent target zone to be filled and the pre-established set of coordinates of the current target zone over which the material distribution end is positioned.

9. The method according to claim 8, wherein the step of generating commands in the alignment module capable of being used by at least one of the receiving vehicle and the transferring vehicle comprises the step of actuating a control mechanism to move the material distribution end over the pre-established set of coordinates of the subsequent target zone to be filled.

10. The method according to claim 8, wherein the step of generating commands in the alignment module capable of being used by at least one of the receiving vehicle and the transferring vehicle comprises the step of changing a lateral offset between the receiving vehicle and the transferring vehicle to move the material distribution end from over the current target zone to over the subsequent target zone to be filled.

11. The method according to claim 10, wherein the step of changing a lateral offset between the receiving vehicle and the transferring vehicle comprises the step of:

establishing a current lateral offset between the receiving vehicle and the transferring vehicle;

converting the pre-established set of coordinates between the current target zone and the subsequent zone to be filled into a change in the current lateral offset to form a new lateral offset;

transmitting the new lateral offset to a steering controller of the receiving vehicle; and

steering the receiving vehicle to create the new lateral offset between the receiving vehicle and the transferring vehicle,

whereby the material distribution end is aligned the over the subsequent target zone to be filled.

12. The method according to claim 8, wherein the step of generating commands in the alignment module capable of being used by at least one of the receiving vehicle and the transferring vehicle comprises the step of changing a fore/aft offset between the receiving vehicle and the transferring vehicle to move the material distribution end from over the current target zone to over the subsequent target zone to be filled.

13. The method according to claim 12, wherein the step of changing a fore/aft offset between the receiving vehicle and the transferring vehicle comprises the step of:

establishing an initial fore/aft offset between the receiving vehicle and the transferring vehicle;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current fore/aft offset to form a new fore/aft offset;

transmitting the new fore/aft offset to a propulsion controller of the receiving vehicle, or a braking controller of the receiving vehicle; and

accelerating or braking the receiving vehicle to create the new fore/aft offset between the receiving vehicle and the transferring vehicle,

whereby the material distribution end is aligned the over the subsequent target zone to be filled.

14. The method according to claim 1, wherein the step of detecting when the current target area of the bin is filled with the material comprises the step of detecting a representation of the fill level or volumetric distribution of the material in the current target area of the bin.

15. The method according to claim 14, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the current target area of the bin further comprises the steps of:

receiving data from a distributed fill state sensor; and

generating the representation of the fill level or volumetric distribution of the material in the current target area of the bin based on the distributed fill state sensor data.

16. The method according to claim 14, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the current target area of the bin further comprises the steps of:

receiving rectified image data from the single imaging device;

generating disparity image data based on the rectified image data; and

generating the representation of the fill level or volumetric distribution of the material in the bin using range data based on the disparity image data.

17. The method according to claim 16, wherein the step of receiving rectified image data from the single imaging device further comprises receiving a rectified image data from an imaging device on the receiving vehicle, wherein there are no imaging devices on the transferring vehicle.

18. The method according to claim 1, wherein the material is agricultural material.

19. The method according to claim 1, wherein the material is a mineral material.

20. The method according to claim 1, further comprising:

sending the commands generated by the alignment module to a master controller,

wherein the master controller is capable of controlling a propulsion controller, a braking controller, and a steering controller of the transferring vehicle,

wherein the master controller is further capable of controlling a propulsion controller, a braking controller, and a steering controller of the receiving vehicle.