Method and stereo vision system for managing the unloading of an agricultural material from a vehicle

• US 8,649,940 B2
• Filed: 03/14/2012
• Issued: 02/11/2014
• Est. Priority Date: 02/10/2012
• Status: Active Grant

• Current Assignee: Deere & Company
• Original Assignee: Deere & Company
• Inventors: Bonefas, Zachary T
• Primary Examiner(s): To, Tuan C.
• Assistant Examiner(s): SMITH, JELANI A
• Application Number:
  US13/419,818
• Publication Number:
  US 20130211675A1
• Time in Patent Office: 699 Days
• Field of Search: 701/50, 701/41, 701/409, 701/28, 701/26, 701/24, 701/23, 701/2, 56/102, 56/102.AR, 460/114, 460/119, 382/100, 382/103, 382/104, 141/1, 141/83, 702/150
• US Class Current: 701/41
• CPC Class Codes: A01B 69/001 Steering by means of optical assistance, e.g. television cameras steering devices for road marking vehicles E01C23/163
• CPC Class Codes: A01B 69/008 automatic
• CPC Class Codes: A01D 43/087 with controllable discharge spout
• CPC Class Codes: B62D 12/02 for vehicles operating in tandem
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593

Abstract

A stereo imaging device on a propelled portion of the receiving vehicle collects image data. A container identification module identifies a container perimeter of a storage portion in the collected image data. A spout identification module is adapted to identify a spout of the harvesting vehicle in the collected image data. An alignment module is adapted to determine the relative position of the spout and the container perimeter and to generate command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central zone or target zone of the container perimeter. A steering controller is associated with a steering system of the propelled portion for steering the receiving vehicle in accordance with the cooperative alignment.

Claims

1. A system for facilitating the transfer of agricultural material from a harvesting vehicle to a receiving vehicle, the system comprising:

a receiving vehicle comprising a propelled portion for propelling the receiving vehicle and a storage portion for storing agricultural material;

a stereo imaging device facing towards the storage portion of the receiving vehicle, the imaging device collecting image data;

a container identification module for identifying a container perimeter of the storage portion in the collected image data, the storage portion having an opening inward from the container perimeter for receipt of the agricultural material;

a spout identification module for identifying a spout of the harvesting vehicle in the collected image data based on color discrimination to identify background pixels from one or more selected spout pixels with associated spout pixel patterns or color attributes on the spout for identification purposes; and

an alignment module for determining the relative position of the spout and the container perimeter and for generating command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central zone of the container perimeter; and

a steering controller associated with a steering system of the propelled portion for steering the receiving vehicle in accordance with the cooperative alignment.

2. The system according to claim 1 wherein the stereo imaging device is mounted on the propelled portion of the receiving vehicle.

3. The system according to claim 1 further comprising:

a user interface for entering dimensional parameters related to the receiving vehicle, the dimensional parameters comprising a distance between a trailer hitch and front wheel rotational axis of the storage portion.

4. The system according to claim 1 wherein the container identification module further comprises:

an edge detector for measuring the strength of one or more edges in the image data;

a linear Hough transformer for identifying an angle and offset of candidate linear segments in the image data; and

a container localizer adapted to use spatial and angular constraints to eliminate candidate linear segments that cannot logically form part of the identified linear segments of the container perimeter, the localizer transforming the identified linear segments into three dimensional coordinates relative to the receiving vehicle and harvesting vehicle.

5. The system according to claim 4 further comprising:

the container localizer receiving measurements of dimensions of the container perimeter or the storage portion of the vehicle to facilitate identification of candidate linear segments that qualify as identified linear segments of the container perimeter.

6. The system according to claim 4 further comprising:

the container localizer receiving an estimated angle of the storage portion relative to the propelling portion of the vehicle to facilitate identification of candidate linear segments that qualify as identified linear segments of the container perimeter.

7. The system according to claim 6 further comprising:

an angle estimator for providing the received estimated angle of the storage portion relative to the propelling portion of the vehicle.

8. The system according to claim 7 wherein the angle estimator comprises a Kalman filter.

9. The system according to claim 1 wherein the spout identification module further comprises:

a spout classifier that is configured to identify candidate pixels in the image data based on the color discrimination and at least one of reflectivity, intensity, or texture features of the image data, where the candidate pixels represent a portion of the spout or combine auger spout.

10. The system according to claim 9 further comprising:

a spout localizer for estimating a relative position or relative angle of the spout to the imaging device based on the classified, identified candidate pixels of a portion of the spout.

11. The system according to claim 9 further comprising:

the spout localizer receiving an estimated combine spout position relative to the mounting location of the imaging device based on previous measurements to provide constraint data on where the spout can be located possibly.

12. The system according to claim 11 further comprising:

a spout position estimator for providing the estimated combine spout position to the spout localizer.

13. The system according to claim 12 wherein the spout position estimator comprises a Kalman filter.

14. The system according to claim 1 further comprising:

a propulsion controller for temporarily increasing a relative speed of the receiving vehicle relative to the transferring vehicle if an image processing module senses that a front volume of the storage portion is currently filled to a target level or until the entire storage portion reaches the target fill level.

15. The system according to claim 1 further comprising:

a propulsion controller for temporarily decreasing a relative speed of the receiving vehicle relative to the transferring vehicle if an image processing module senses that a rear volume of the storage portion is currently filled to a target level or until the entire storage portion reaches the target fill level.

16. A method for facilitating the transfer of agricultural material from a harvesting vehicle to a receiving vehicle, the method comprising:

propelling a storage portion of a receiving vehicle for storing agricultural material;

collecting imaging data by a first imaging device on the harvesting vehicle or the receiving vehicle;

identifying a container perimeter of the storage portion in the collected image data, the storage portion having an opening inward from the container perimeter for receipt of the agricultural material;

identifying a spout of the harvesting vehicle in the collected image data based on color discrimination to identify background pixels from one or more selected spout pixels with associated spout pixel patterns or color attributes on the spout for identification purposes; and

determining the relative position of the spout and the container perimeter and for generating command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central zone of the container perimeter; and

steering the receiving vehicle in accordance with the cooperative alignment.

17. The method according to claim 16 further comprising:

entering, via a user interface, dimensional parameters related to the receiving vehicle, the dimensional parameters comprising a distance between a trailer hitch and front wheel rotational axis of the storage portion.

18. The method according to claim 16 wherein identifying the container perimeter further comprises:

measuring a strength of one or more edges in the image data;

identifying an angle and offset of candidate linear segments in the image data;

using spatial and angular constraints to eliminate candidate linear segments that cannot logically form part of the identified linear segments of the container perimeter; and

transforming the identified linear segments into three dimensional coordinates relative to the receiving vehicle and harvesting vehicle.

19. The method according to claim 18 further comprising:

receiving measurements of dimensions of the container perimeter or the storage portion of the vehicle to facilitate the identification of candidate linear segments that qualify as the identified linear segments of the container perimeter.

20. The method according to claim 18 further comprising:

receiving an estimated angle of the storage portion relative to the propelling portion of the vehicle to facilitate the identification of candidate linear segments that qualify as the identified linear segments of the container perimeter.

21. The method according to claim 20 further comprising:

providing the received estimated angle of the storage portion relative to the propelling portion of the vehicle.

22. The method according to claim 16 wherein identifying the spout identification module further comprises:

identifying candidate pixels in the image data based on the color discrimination and at least one of reflectivity, intensity or texture features of the image data, where the candidate pixels represent a portion of the spout or combine auger spout.

23. The method according to claim 22 further comprising:

estimating a relative position of the spout to the imaging device based on the classified, identified candidate pixels of a portion of the spout.

24. The method according to claim 22 further comprising:

receiving an estimated combine spout position relative to the mounting location of the imaging device, or a spout angle with reference to a pivot point of the spout, based on previous measurements to provide constraint data on where the spout can be located possibly.

25. The method according to claim 24 further comprising:

providing the estimated combine spout position to a spout localizer.

26. The method according to claim 16 further comprising:

temporarily increasing a relative speed of the receiving vehicle relative to the transferring vehicle if an image processing module senses that a front volume of the storage portion is currently filled to a target level or until the entire storage portion reaches the target fill level.

27. The method according to claim 16 further comprising:

temporarily decreasing a relative speed of the receiving vehicle relative to the transferring vehicle if an image processing module senses that a rear volume of the storage portion is currently filled to a target level or until the entire storage portion reaches the target fill level.

28. A system for facilitating the transfer of agricultural material from a harvesting vehicle to a receiving vehicle, the system comprising:

a receiving vehicle comprising a propelled portion for propelling the receiving vehicle and a storage portion for storing agricultural material;

a stereo imaging device mounted on the receiving vehicle and facing towards the storage portion of the receiving vehicle, the imaging device collecting image data;

a container identification module for identifying a container perimeter of the storage portion in the collected image data, the storage portion having an opening inward from the container perimeter for receipt of the agricultural material;

a spout identification module for identifying a spout of the harvesting vehicle in the collected image data; and

an alignment module for determining the relative position of the spout and the container perimeter and for generating command data to the propelled portion to steer the storage portion in cooperative alignment such that the spout is aligned within a central zone of the container perimeter; and

a steering controller associated with a steering system of the propelled portion for steering the receiving vehicle in accordance with the cooperative alignment, wherein the container identification module, the spout identification module, the alignment module, and the steering controller are on the receiving vehicle.