System and method of material handling using one or more imaging devices on the transferring vehicle and on the receiving vehicle to control the material distribution into the storage portion of the receiving vehicle

• US 9,463,939 B2
• Filed: 02/11/2013
• Issued: 10/11/2016
• Est. Priority Date: 02/10/2012
• Status: Active Grant

• Current Assignee: Deere & Company
• Original Assignee: Deere & Company
• Inventors: Bonefas, Zachary T., Bartholomew, Darin E.
• Primary Examiner(s): MAUST, TIMOTHY LEWIS
• Application Number:
  US14/377,409
• Publication Number:
  US 20160009509A1
• Time in Patent Office: 1,338 Days
• Field of Search: 701/50, 141/1, 141/2, 141/94, 141/95, 141/98, 141/192, 141/198
• US Class Current: 1/1
• CPC Class Codes: A01D 43/087 with controllable discharge spout
• CPC Class Codes: B60P 1/42 mounted on the load-transporting element
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B62D 6/001 the torque NOT being among the input parameters
• CPC Class Codes: B65B 1/48 Checking volume of filled material
• CPC Class Codes: B65G 67/22 Loading moving vehicles
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0293 Convoy travelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

First imaging device collects first image data, whereas second imaging device collects second image data of a storage portion. A container identification module identifies a container perimeter of the storage portion in at least one of the collected first image data and the collected second image data. A spout identification module is adapted to identify a spout of the transferring vehicle in the collected image data. An image data evaluator determines whether to use the first image data, the second image data, or both based on an evaluation of the intensity of pixel data or ambient light conditions. An alignment module is adapted to determine the relative position of the spout and the container perimeter and to generate command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central zone or a target zone of the container perimeter.

Claims

1. A method for facilitating the transfer of material from a transferring vehicle having a material distribution end to a receiving vehicle having a bin to the store transferred material, the method comprising the steps of:

a. identifying and locating the bin using at least one of first image data and second image data,

wherein a first imaging device on the receiving vehicle captures the first image data comprising a field of view containing the material distribution end and the bin,

wherein a second imaging device on the transferring vehicle captures the second image data comprising a field of view containing the material distribution end and the bin;

b. detecting a representation of the fill level or volumetric distribution of the material in the bin;

c. aligning the material distribution end over a current target area of the bin requiring the material;

d. determining subsequent target areas of the bin that require material based on the representation of the fill level or volumetric distribution of the material in the bin;

e. transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle;

f. detecting when the current target area of the bin is filled with the material;

g. repeating steps c-f until the subsequent target areas of the bin are filled; and

h. terminating the transfer of the material from the transferring vehicle to the receiving vehicle.

2. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is one-dimensional.

3. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is two-dimensional.

4. The method according to claim 1, wherein the representation of the fill level or volumetric distribution of the material in the bin is three-dimensional.

5. The method according to claim 1, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the bin further comprises the steps of:

receiving data from a distributed fill state sensor; and

generating the representation of the fill level or volumetric distribution of the material in the bin based on the distributed fill state sensor.

6. The method according to claim 1, wherein the step of detecting the representation of the fill level or volumetric distribution of the material in the bin further comprises the steps of:

receiving rectified image data from at least one of the first imaging device and the second imaging device;

generating disparity image data based on the rectified image data; and

generating the representation of the fill level or volumetric distribution of the material in the bin using range data based on the disparity image data.

7. The method according to claim 1, wherein the step of determining portions of the bin that require material based on the representation of the fill level or volumetric distribution of the material in the bin comprises the steps of:

developing a target zone matrix of the bin, wherein each target zone of the matrix is identified by a pre-established set of coordinates related to the bin;

identifying target zones of the matrix that are filled and not filled based on the representation of the fill level or volumetric distribution of the material in the bin; and

determining the pre-established set of coordinates of a subsequent target zone to be filled and the pre-established set of coordinates of the current target zone over which the material distribution end is positioned.

8. The method according to claim 7, wherein the step of aligning the material distribution end over a current target area of the bin requiring the material comprises the step of actuating a control mechanism to move the material distribution end over the pre-established set of coordinates of a subsequent target zone to be filled.

9. The method according to claim 8, wherein the step of actuating a control mechanism to move the material distribution end over the pre-established set of coordinates of a subsequent target zone to be filled comprises the steps of:

establishing a current lateral spout angle of the material distribution end;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current lateral spout angle to form a new lateral spout angle;

transmitting the new lateral spout angle to a spout controller of the transferring vehicle; and

actuating the spout controller from the current lateral spout angle to the new lateral spout angle,

whereby the material distribution end is aligned the over the subsequent target zone to be filled.

10. The method according to claim 7, wherein the step of aligning the material distribution end over a current target area of the bin requiring the material comprises the step of changing a linear spatial separation between the receiving vehicle and the transferring vehicle to move the material distribution end from over the current target zone to over the subsequent target zone to be filled.

11. The method according to claim 10, wherein the step of changing a linear spatial separation between the receiving vehicle and the transferring vehicle comprises the step of:

establishing a current linear spatial separation between the receiving vehicle and the transferring vehicle;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current linear spatial separation to form a new linear spatial separation;

transmitting the new lateral offset to a steering controller of the transferring vehicle; and

steering the transferring vehicle to create the new linear spatial separation between the receiving vehicle and the transferring vehicle,

whereby the material distribution end is aligned the over the subsequent target zone to be filled.

12. The method according to claim 7, wherein the step of aligning the material distribution end over a current target area of the bin requiring the material comprises the step of changing a linear spatial separation between the receiving vehicle and the transferring vehicle to move the material distribution end from over the current target zone to over the subsequent target zone to be filled.

13. The method according to claim 12, wherein the step of changing a linear spatial separation between the receiving vehicle and the transferring vehicle comprises the step of:

establishing an initial linear spatial separation between the receiving vehicle and the transferring vehicle;

converting the pre-established set of coordinates between the current target zone and the subsequent target zone to be filled into a change in the current linear spatial separation to form a new linear spatial separation;

transmitting the new linear spatial separation to a propulsion controller of the transferring vehicle or the receiving vehicle or both, or a braking controller of the transferring vehicle or the receiving vehicle or both; and

accelerating or braking the transferring vehicle or the receiving vehicle or both to create the new linear spatial separation between the receiving vehicle and the transferring vehicle,

whereby the material distribution end is aligned the over the subsequent target zone to be filled.

14. The method according to claim 1, wherein the step of transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle comprises the step of actuating an auger.

15. The method according to claim 1, wherein the step of transferring the material from the transferring vehicle to the current target area of the bin of the receiving vehicle comprises the step of actuating a spout.

16. The method according to claim 1, wherein the step of detecting when the current target area of the bin is filled with the material comprises the step of detecting a representation of the fill level or volumetric distribution of the material in the current target zone of the bin.

17. The method according to claim 16, wherein the step of detecting a representation of the fill level or volumetric distribution of the material in the current target zone of the bin further comprises the steps of:

receiving data from a distributed fill state sensor; and

generating the representation of the fill level or volumetric distribution of the material in the current target zone of the bin based on the distributed fill state sensor data.

18. The method according to claim 16, wherein the step of detecting a representation of the fill level or volumetric distribution of the material in the current target zone of the bin further comprises the steps of:

receiving rectified image data from at least one of the first imaging device and the second imaging device;

generating disparity image data based on the rectified image data; and

generating the representation of the fill level or volumetric distribution of the material in the bin using range data based on the disparity image data.

19. The method according to claim 1, wherein the step of terminating the transfer of the material from the transferring vehicle to the receiving vehicle when the current target zone of the bin is filled comprises the step of stopping the auger.

20. The method according to claim 1, wherein the step of terminating the transfer of the material from the transferring vehicle to the receiving vehicle when the current target zone of the bin is filled comprises the step of stopping the spout.

21. The method according to claim 1, wherein the material is agricultural material.

22. The method according to claim 1, wherein the material is a mineral material.

23. The method according to claim 6, wherein the step of receiving rectified image data from at least one of the first imaging device and the second imaging device further comprises receiving a rectified image data from the first imaging device.

24. The method according to claim 6, wherein the step of receiving rectified image data from at least one of the first imaging device and the second imaging device further comprises:

receiving a first rectified image data from the first imaging device;

receiving a second rectified image data from the second imaging device; and

using an image data evaluator to determine whether to use the first rectified image data, the second rectified image data, or both the first rectified image data and the second rectified image data.

25. The method according to claim 18, wherein the step of receiving rectified image data from at least one of the first imaging device and the second imaging device further comprises receiving a rectified image data from the first imaging device.

26. The method according to claim 18, wherein the step of receiving rectified image data from at least one of the first imaging device and the second imaging device further comprises:

receiving a first rectified image data from the first imaging device;

receiving a second rectified image data from the second imaging device; and

using an image data evaluator to determine whether to use the first rectified image data, the second rectified image data, or both the first rectified image data and the second rectified image data.