System and method of material handling using one imaging device on the receiving vehicle to control the material distribution into the storage portion of the receiving vehicle

• US 9,511,958 B2
• Filed: 02/11/2013
• Issued: 12/06/2016
• Est. Priority Date: 02/10/2012
• Status: Active Grant

• Current Assignee: Deere & Company
• Original Assignee: Deere & Company
• Inventors: Bonefas, Zachary T.
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Lee, Jonathan S
• Application Number:
  US14/377,402
• Publication Number:
  US 20150356722A1
• Time in Patent Office: 1,394 Days
• US Class Current: 1/1
• CPC Class Codes: A01D 43/087 with controllable discharge spout
• CPC Class Codes: B60P 1/42 mounted on the load-transporting element
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B62D 6/001 the torque NOT being among the input parameters
• CPC Class Codes: B65B 1/48 Checking volume of filled material
• CPC Class Codes: B65G 67/22 Loading moving vehicles
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0293 Convoy travelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A single imaging device collects image data of a storage portion. A container module identifies a container perimeter of the storage portion in at least one of the collected image data. A spout module is adapted to identify a spout of the transferring vehicle in the collected image data. An arbiter determines whether to use the image data based on an evaluation of the intensity of pixel data or ambient light conditions. An alignment module is adapted to determine the relative position of the spout and the container perimeter and to generate command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central or target zone of the container perimeter.

Claims

1. A system for facilitating the transfer of material from a transferring vehicle to a receiving vehicle, the system comprising:

a receiving vehicle comprising a propelled portion for propelling the receiving vehicle and a storage portion for storing material;

a single imaging device facing towards the storage portion of the receiving vehicle, the imaging device collecting image data;

a container identification module for identifying a container perimeter of the storage portion, wherein the container identification module identifies the container perimeter by locating a linear set of pixels associated with the container perimeter in the collected image data;

a spout identification module for identifying a spout of the transferring vehicle, wherein the spout identification module identifies the spout by locating pixels associated with the spout in the collected image data;

an image data evaluator for determining whether to use the collected image data for alignment of a relative position of the spout and the container perimeter based on evaluation of material variation of intensity of pixel data or material variation in ambient light conditions during a sampling time interval;

an alignment module for determining the relative position of the spout and the container perimeter using the image data and for generating command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a target zone of the container perimeter; and

a steering controller associated with a steering system of the propelled portion for steering the receiving vehicle in accordance with the cooperative alignment.

2. The system according to claim 1, wherein the imaging device is mounted on the receiving vehicle, wherein there are no imaging devices on the transferring vehicle.

3. The system according to claim 1, wherein the imaging device has a field of view of the storage portion.

4. The system according to claim 1, wherein the imaging device comprises a stereo vision camera.

5. The system according to claim 1, further comprising an optical sensor associated with the imaging device, the image data evaluator deciding to use the image data if the variation in ambient light over a sampling time interval is less than or equal to a maximum ambient light variation, as measured by the optical sensor.

6. The system according to claim 1, further comprising an image processing module associated with the imaging device, the image data evaluator deciding to use the image data if the variation in pixel intensity of a spout or spout end in the image data over a sampling time interval is less than or equal to a maximum pixel intensity variation, as detected by the image processing module.

7. The system according to claim 1, further comprising the image data evaluator adapted for determining whether to use the image data for the identification of the container perimeter and the identifying of the spout, based on pixel intensity in rejected image data being outside of a desired range or variation in pixel intensity during the sampling time interval, where the image data evaluator is configured to disable selectively the processing or use of the rejected image data that comprises a portion of the collected first image data that would otherwise be corrupted by excessive transient sunlight during sunrise or sunset.

8. The system according to claim 1, wherein the imaging device has its optical axis, perpendicular to its lens, tilted downward from a generally horizontal plane.

19. The system according to claim 1, wherein the material is agricultural material.

20. The system according to claim 1, wherein the material is a mineral material.

9. A method for facilitating the transfer of material from a transferring vehicle to a receiving vehicle, the method comprising the steps of:

collecting image data by an imaging device facing towards a storage portion of a receiving vehicle, the storage portion capable of storing material;

identifying a container perimeter of the storage portion by locating a linear set of pixels associated with the container perimeter in the collected image data;

identifying a spout of the transferring vehicle by locating pixels associated with the spout in the collected image data;

determining whether to use the image data for alignment of a relative position of the spout and the container perimeter based on evaluation of material variation of intensity of pixel data or material variation in ambient light conditions during a sampling time interval;

determining the relative position of the spout and the container perimeter using the image data; and

generating command data for a propelled portion of the receiving vehicle to steer the storage portion in cooperative alignment such that the spout is aligned within a target zone of the container perimeter.

10. The method according to claim 9, further comprising the step of transmitting a data message for steering the receiving vehicle in accordance with the cooperative alignment.

11. The method according to claim 9, wherein the imaging device has a first field of view of the storage portion.

12. The method according to claim 9, wherein the imaging device comprises a stereo imaging device, and further comprising the step of creating a stereo image from the collected image data with reference to the relative position and orientation of the imaging device.

13. The method according to claim 9, further comprising the step of deciding to use the image data if the variation in ambient light over a sampling time interval is less than or equal to a maximum ambient light variation, as measured by an optical sensor associated with the imaging device.

14. The method according to claim 9, further comprising the step of deciding to use the image data if the variation in pixel intensity of a spout or spout end in the image over a sampling time interval is less than or equal to a maximum pixel intensity variation.

15. The method according to claim 9, further comprising the step of determining whether to use the first image data for the identification of the container perimeter and the identifying of the spout, based on pixel intensity in rejected image data being outside of a desired range or variation in pixel intensity during the sampling time interval using an image data evaluator, where the image data evaluator is configured to disable selectively the processing or use of rejected image data that comprises a portion of the collected image data that would otherwise be corrupted by excessive transient sunlight during sunrise or sunset.

21. The method according to claim 9, wherein the material is agricultural material.

22. The method according to claim 9, wherein the material is a mineral material.

16. A system for facilitating the transfer of material from a transferring vehicle to a receiving vehicle, the system comprising:

a receiving vehicle comprising a propelled portion for propelling the receiving vehicle and a storage portion for storing material;

an imaging device facing towards the storage portion of the receiving vehicle, the imaging device collecting image data;

a container identification module for identifying a container perimeter of the storage portion, wherein the container identification module identifies the container perimeter by locating a linear set of pixels associated with the container perimeter in the collected image data;

a spout identification module for identifying a spout of the transferring vehicle, wherein the spout identification module identifies the spout by locating pixels associated with the spout in the collected image data;

an adjustable mast capable of adjusting a height of the imaging device within a height range and adjusting a down-tilt angle of the imaging device within a down-tilt angular range;

an image processing module for determining whether to adjust the height of the imaging device or whether to decrement or increment the down-tilt angle of the imaging device based on evaluation of material variation of intensity of pixel data or material variation in ambient light conditions during a sampling time interval;

an alignment module for determining the relative position of the spout and the container perimeter using the image data and for generating command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a target zone of the container perimeter; and

a steering controller associated with a steering system of the propelled portion for steering the receiving vehicle in accordance with the cooperative alignment.

17. The system according to claim 16, wherein the image processing module sends a data message to a mast controller to increment or decrement the down-tilt angle if the material variation of intensity of pixel data or if the material variation in ambient light conditions exceeds a threshold variation level during a sampling time interval.

18. The system according to claim 16, wherein the image processing module sends a data message to a mast controller to increment or decrement the down-tilt angle at discrete levels within an angular range of approximately negative ten degrees to approximately negative twenty-five degrees from a generally horizontal plane.

23. The system according to claim 16, wherein the material is agricultural material.

24. The system according to claim 16, wherein the material is a mineral material.