Agricultural harvester with robotic control

• US 6,336,051 B1
• Filed: 08/28/2000
• Issued: 01/01/2002
• Est. Priority Date: 04/16/1997
• Status: Expired due to Fees

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Whittaker, William, Fitzpatrick, Kerien, Pangels, Henning, Pilarski, Thomas, Happold, Michael, Ollis, Mark, Stentz, Anthony
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): Backer, Firmin
• Application Number:
  US09/648,741
• Time in Patent Office: 491 Days
• Field of Search: 700/95, 700/245, 700/209, 701/50, 701/213, 701/207, 701/208, 701/23, 318/568
• US Class Current: 700/50
• CPC Class Codes: A01B 79/005 Precision agriculture
• CPC Class Codes: A01D 41/1278 for automatic steering
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

An agricultural harvesting machine (harvester) is automatically controlled and steered so as to completely harvest all crop in a field. A Global Positioning System (GPS) and an Inertial Navigation System provide data regarding the position and orientation of the harvester in a field. A Field Coverage Planner constructs a field coverage plan which defines a path for the harvester to follow in cutting crop from a field. Video images of the field in front of the harvester are derived alternately from two video cameras mounted on opposite sides of the harvester. The video images are processed in a Video Processing Computer to detect the crop line between cut and uncut crop and generate steering signals for steering the harvester along the crop line. Video images are also processed to detect the end of a row at the end of a field and supply a measure of the distance to the end of the row. When the end of a row is reached, the harvester executes a 180° spin turn and traverses the field in the opposite direction.

Claims

1. A robotic control system for an agricultural harvester, the robotic control system comprising an execution monitor module which is sequenced through a plurality of finite states by trigger messages from a field coverage planner module, a global trajectory tracker module, a controller module, an end of row detector module and a field coverage monitor module, the execution monitor module activating at least one of the field coverage planner module, the global trajectory tracker module, the end of row detector module, a crop line tracker module or a local trajectory tracker module in each finite state so as to generate steering signals for steering the harvester along a path specified by a field coverage plan to cut all crop in a field.

2. A robotic control system as claimed in claim 1 wherein said global trajectory module generates steering votes indicating the preferred direction the harvester should be steered so as to follow said path and said crop line tracker module generates steering votes on the direction the harvester should be steered to follow a crop line between cut and uncut crop, the system further comprising a continuously running steering arbiter module for developing steering commands from all the steering votes and a continuously running controller responsive to the steering commands for developing signals to differentially drive front wheels of the harvester to steer it.

3. A robotic control system as claimed in claim 1 wherein said crop line tracker module generates, for each image, steering votes on the direction the harvester should be steered to follow a crop line between cut and uncut crop, the system further comprising a continuously running steering arbiter module for developing a steering command from all the steering votes, and a continuously running controller responsive to the steering commands for developing signals to differentially drive front wheels of the harvester to steer it.

4. A robotic control system as claimed in claim 3 wherein, in developing the steering commands, the votes from the crop line tracker module are weighted differently from the votes from the global trajectory tracker module.

5. A robotic control system as claimed in claim 1 and further comprising at least one video camera mounted on the harvester for viewing the changing scene in front of the harvester and producing images thereof as the harvester moves along the planned path, the crop line tracker module compensating the images for shadows in the scene and deriving, from each compensated image, steering votes for steering the harvester along a crop line between cut and uncut crop.

6. A robotic control system as claimed in claim 5 wherein said steering votes are time decayed over N images to produce a steering preference signal.

7. A robotic control system as claimed in claim 1 and further comprising at least one video camera mounted on the harvester for viewing the changing scene in front of the harvester and producing images of the changing scene as the harvester moves along the planned path, said end of row detector module scanning each line of an image to determine if a crop line boundary is indicated in the line, and, from the determination of which lines indicate a crop line boundary and which lines do not, determining the most likely line imaging the end of a crop row.

8. A robotic control system for a harvester, including at least one video camera mounted on the harvester for viewing the changing scene in front of the harvester and producing images of the changing scene as the harvester moves along the planned path, and a crop line tracker module for analyzing each of the images according to an adaptively updated color discriminant function to derive steering votes for steering the harvester along a crop line between cut and uncut crop.

9. A robotic control system as claimed in claim 8 wherein said crop line tracker module includes:

a discriminator for analyzing each pixel of each line of each produced image and assigning a discriminant value to each said pixel; and

a segmenter for computing a best fit step function for the pixels of each line and defining a selected pixel in each line to represent the location of the crop line.

10. A robotic control system as claimed in claim 8 wherein, after each image is analyzed, the crop line tracker module computes the Fisher linear discriminant in RGB space between the cut and uncut crop as the discriminant for the next image.

11. A robotic control system for a harvester, including at least one video camera mounted on the harvester for viewing the changing scene in front of the harvester and producing images of the changing scene as the harvester moves along the planned path, and an end of row detector module scanning each line of an image to determine if a crop line boundary is indicated in the line, and, from the determination of which lines indicate a crop line boundary and which lines do not, determining the most likely line imaging the end of a crop row.

12. A robotic control system as claimed in claim 11 wherein said end of row detector module computes a computed distance from a current harvester position to determined position corresponding to a most likely end of row position, said computed distance being compared with a stored distance value, said end of row detector module processing a new image and determining a new determined position from which a new computed distance is computed whenever said computed distance is greater than said stored distance.

13. A robotic control system as claimed in claim 12 wherein said end of row detector module continues to process new images and determine new determined positions until the new computed distance is less than or equal to said stored distance for a predetermined number of computed new determined positions.

14. A method of harvesting crop material with a crop harvesting machine having a robotic control system, comprising the steps of:

providing an execution monitor module which is sequenced through a plurality of finite states by trigger messages from a field coverage planner module, a global trajectory tracker module, a controller module, an end of row detector module and a field coverage monitor module; and

activating at least one of the field coverage planner module, the global trajectory tracker module, the end of row detector module, a crop line tracker module or a local trajectory tracker module in each finite state so as to generate steering votes for steering the harvester along a path specified by a field coverage plan to cut all crop in a field.

15. A method of harvesting crop material as claimed in claim 14, further comprising the steps of:

generating steering votes from at least said global trajectory module, which indicates the preferred direction the harvester should be steered so as to follow said path, and said crop line tracker module which defines a crop line between cut and uncut crop along which the harvester should be steered;

arbitrating said steering votes by a continuously running steering arbiter module for developing steering commands from all the steering votes and a continuously running controller responsive to the steering commands for developing signals to differentially drive front wheels of the harvester to steer it; and

steering said crop harvesting machine in response to said steering commands.

16. A method of harvesting crop material as claimed in claim 15, further comprising the steps of:

weighting said steering votes in said arbitrating step such that said steering votes from said crop line tracker module are weighted differently than the steering votes from said global trajectory tracker module.

17. A method of harvesting crop material as claimed in claim 16, further comprising the steps of:

producing images from at least one video camera mounted on said harvesting machine to be indicative of the changing scene in front of said harvesting machine as said harvesting machine moves along a steering path;

compensating the images for shadows to create compensated images; and

deriving from said compensated images said steering votes for said crop line tracking module to steer said harvesting machine along a crop line between cut and uncut crop material.

18. A method of harvesting crop material as claimed in claim 17 wherein said deriving step includes the step of:

analyzing each of said images according to an adaptively updated color discriminant function to define said crop line between cut and uncut crop material.

19. A method of harvesting crop material as claimed in claim 17, further comprising the steps of:

scanning each line of each image by said end of row detector module to determine if a crop line boundary is indicated; and

determining from an analysis of each scanned line the most likely line imaging the end of a crop row.