Vision-based crop line tracking for harvesters

• US 5,911,669 A
• Filed: 04/16/1997
• Issued: 06/15/1999
• Est. Priority Date: 04/19/1996
• Status: Expired due to Fees

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Hoffman, Regis, Ollis, Mark, Whittaker, William, Fitzpatrick, Kerien, Stentz, Anthony
• Primary Examiner(s): Carone, Michael J.
• Assistant Examiner(s): PEZZUTO, ROBERT ERIC
• Application Number:
  US08/840,771
• Time in Patent Office: 790 Days
• Field of Search: 56/10.2 A, 56/10.2 F, 56/10.2 R, 56/10.2 J, 56/11.9, 56/16.7, 56/DIG. 15, 172/2, 172/4, 172/4.5, 364/413.13, 364/424.07, 382/10, 382/11, 382/12, 382/13, 382/14, 382/15
• US Class Current: 56/10.20F
• CPC Class Codes: A01B 69/001 Steering by means of optical assistance, e.g. television cameras steering devices for road marking vehicles E01C23/163
• CPC Class Codes: A01D 41/1278 for automatic steering
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A self-propelled harvester is provided with apparatus for tracking a crop line and steering the harvester along the crop line without operator assistance. The apparatus includes a video camera for viewing the landscape in front of the harvester and forming an image which may be electronically scanned pixel-by-pixel and line by line to produce a video signal containing information regarding the spectral characteristics of the image at each pixel. A microprocessor is programmed to act as a discriminator and a segmentor. The discriminator produces discriminant signals D(i,j) where i is the magnitude of a discriminant function and j is the relative position of a pixel on a scan line. The segmentor determines a step function which best fits the magnitudes of the discriminant signals for one scan line. The pixel position j_n corresponding to the step of this step function is taken as a vote on the location of the crop line and thus a vote on the direction the harvester should be steered. The votes are summed for each pixel position j over one full scan of image. The summed votes are then normalized, time-weighted, and averaged with the normalized and time-weighted votes derived from previously scanned images to derive a steering command.

Claims

1. A method of steering an agricultural harvester along a crop line, the harvester having control means for steering wheels to thereby steer the harvester, said method comprising:

a) mounting on said harvester a viewing means oriented to view and form an image of a landscape, including a crop line, in front of the harvester;

b) scanning at least a window of the image pixel-by-pixel and line-by-line to produce information signals representing said image;

c) generating, from the information signals, a plurality of discriminant function signals D(i,j) for each scan line, where j is the relative position of a pixel on a scan line and i is the magnitude of a discriminant function for pixel j;

d) determining, for each scan line, a pixel position j_n representing the pixel position of a step in a step function which best fits the magnitudes of the discriminant function signals produced for the scan line;

e) developing a sum of the number of times the step function occurs at each pixel position during a complete scan of the window to develop, for that window, vote counts indicating steering direction preferences;

f) repeating steps b) through e); and

,

g) applying to the control means a steering signal determined from the vote counts developed for a plurality of windows.

2. A method as claimed in claim 1 wherein, in step e), the vote counts are normalized and in step g) the steering signal is determined from the normalized vote counts.

3. A method as claimed in claim 1 wherein, in step e), the vote counts are developed by assigning groups of adjacent pixel locations to bins, determining, for each bin, a bin sum representing the number of times the step of the step function occurs at any pixel location assigned to the bin, and normalizing the bin sums to obtain said vote counts.

4. A method as claimed in claim 1 wherein, during step g), the vote counts developed for a plurality of windows are averaged to determine the steering signal applied to the control means.

5. A method as claimed in claim 1 wherein, in step g), the vote counts developed for a plurality of windows are time weighted and then averaged to determine the steering signal applied to the control means.

6. A method as claimed in claim 1 wherein, in step e), the vote counts are developed by assigning groups of adjacent pixel locations to bins, determining, for each bin, a bin sum representing the number of times the step of the step function occurs at any pixel location assigned to the bin, and normalizing the bin sums to obtain said vote counts, and wherein during step g), the vote counts developed for a plurality of windows are time weighted and then averaged to determine the steering signal applied to the control means.

7. A method as claimed in claim 1 wherein step a) comprises mounting a color video camera on the harvester.

8. A method as claimed in claim 1 wherein step a) comprises mounting a RGB video camera on the harvester.

9. A method as claimed in claim 1 wherein step a) comprises mounting on the harvester a black and white video camera having a plurality of associated band pass filters.

10. A method as claimed in claim 1 wherein said discriminant function is the ratio between two given spectral bands.

11. A method as claimed in claim 1 wherein said discriminant function is the ratio of red to green color.

12. A method as claimed in claim 1 wherein said discriminant function is the percentage intensity within a given spectral band.

13. A self-propelled agricultural harvester having control means for steering wheels to thereby steer the harvester, said harvester further comprising:

a viewing means mounted on the harvester and oriented to view and form images of a landscape, including a crop line, in front of the harvester, the viewing means having associated therewith a scanning means for scanning at least a window of an image pixel-by-pixel and line-by-line to produce information signals representing said image;

first means for generating, from the information signals, a plurality of discriminant function signals D(i,j) for each scan line, where j is the relative position of a pixel on a scan line and i is the magnitude of a discriminant function for pixel j;

second means, responsive to said first means, for determining, for each scan line, a pixel position j_n representing the pixel position of a step in a step function which best fits the magnitudes of the discriminant function signals produced for the scan line;

a plurality of counters, there being at least one counter corresponding to each pixel position of the window on one scan line, said counters being responsive to said second means for accumulating totals of the number of times said step occurs at each pixel position during a complete scan of one image, said totals representing vote counts indicating steering direction preferences; and

,

third means, responsive to said vote counts, for applying to the control means a steering signal determined from the vote counts.

14. A harvester as claimed in claim 13 wherein said third means comprises means for averaging the vote counts produced by said counters during scanning of successive images in said window.

15. A harvester as claimed in claim 14 wherein said third means further comprises means for time-weighting the vote counts produced for successive images prior to averaging them so that the vote counts produced for more recent images have greater weight in determining said steering signal.

16. A harvester as claimed in claim 13 wherein said third means includes accumulator means for accumulating bin vote counts by totaling the vote counts produced by said counters for groups of adjacent pixel positions.

17. A harvester as claimed in claim 16 wherein said third means further comprises means for averaging the bin vote counts accumulated by said accumulator means for successive images in said window.

18. A harvester as claimed in claim 17 wherein said third means further comprises means for time-weighting the bin vote counts produced for successive images prior to averaging them so that the bin vote counts produced for more recent images have greater weight in determining said steering signal.

19. A harvester as claimed in claim 13 wherein said viewing means comprises a color video camera and said first means generates discriminant function signals having magnitudes determined by the ratio of two colors at each pixel position.

20. A harvester as claimed in claim 13 wherein said viewing means is a color video camera and said first means generates discriminant function signals having magnitudes determined by the percentage intensity within a given spectral band.

21. A harvester as claimed in claim 13 wherein said control means for steering said wheels comprises first and second drive motors for driving a first and a second of said drive wheels, respectively at different rotational speeds to steer said harvester.