Control System for unmanned automotive vehicle

• US 5,208,750 A
• Filed: 06/15/1988
• Issued: 05/04/1993
• Est. Priority Date: 06/17/1987
• Status: Expired due to Term

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Hattori, Akira, Noso, Kazunori, Kurami, Kunihiko, Kishi, Norimasa
• Primary Examiner(s): Trans, Vincent N.
• Application Number:
  US07/206,667
• Time in Patent Office: 1,784 Days
• Field of Search: 364/424.02, 340/988, 340/995, 340/996, 180/168, 358/103, 358/107
• US Class Current: 701/28
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Stereo image data depicting the lane on which the vehicle is running is rectified into a planar image on which one or more level lines can be generated in a manner to intersect one or both of the edges of the lane. Angles are measured between lines which extend from these the points if intersection in directions normal to the level line or lines and lines which extend tangentially to any curvature which occurs in the road edges at the point of intersection. These angles are recorded, the difference therebetween determined and compared in a manner which determines the attitude of the vehicle on the road. The curvature of the road and the vehicle speed are determined and the best steering angle for the road wheels determined in accordance With a schedule which is recorded in terms of attitude and a parameter variable with the vehicle speed and road curvature.

Claims

1. In a method of controlling an unmanned vehicle, the steps of:

observing a road using camera means;

producing an image of the road based on the output of the camera means;

rectifying the image in a manner which depicts the left and right hand edges of the lane n which the vehicle is running and which enables data to be read therefrom;

determining the curvature of the road and the direction in which it is curving;

determining the attitude of the vehicle on the road by determining the angle between the longitudinal axis and an imaginary line which runs along the middle of the lane depicted in the rectified image;

determining the vehicle speed;

determining a variable which varies as a function of the road curvature, the distance of the center of the vehicle from a side of the lane on which the vehicle is running and the vehicle speed;

performing a table look-up on a selected map using the attitude angle and the variable, to derive a steering angle; and

controlling the angle of the steering wheels of the vehicle with respect to the longitudinal axis of the vehicle to the derived steering angle.

2. A method as claimed in claim 1 wherein said step of curvature determination comprises:

establishing a measurement level line on the rectified image at a location representative of a predetermined distance ahead of said vehicle;

determining first and second points intersect at the locations where said level lines the left and right hand edges of the lane, respectively;

determining the distance from the point that the longitudinal axis of the vehicle intersects said level line to the left hand edge of the lane;

determining the distance form the point that the longitudinal axis of the vehicle intersects said level line to the left hand edge of the lane;

generating first and second lines which pass through said said point and which extend parallel to the longitudinal axis of the vehicle and normal to said level; and

generating third and fourth lines which pass through said first and second points and which extend tangentially with respect to any curvature which exists in the left and the road lane edges at said first and second points, respectively; and

measuring first and second angles, said first angle being defined between said fiesta and third lines and said second angle being defined between said second and fourth lines.

3. A method as claimed in claim 2 further comprising the steps of:

comparing said first angle, said second angle and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

4. A method as claimed in claim 3 wherein said comparing step includes assigning polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are defined on the right of said first and second lines are given a second polarity.

5. A method as claimed in claim 4 wherein the first and second angle data is deemed erroneous in the event that the polarities of said first and second angles are different.

6. A method as claimed in claim 1 wherein said step of curvature determination comprises:

establishing first and second level lines on the rectified image, said first and second level lines being drawn on said image at first and second distances ahead of said vehicle, said second distance being grater than the first distance;

determining first and second points on a selected one of the left and right hand lane edges, where said first and second level lines intersect the same;

generating first and second lines which pass through said first and second points and which are normal to said first and second level lines respectively and parallel with the longitudinal axis of said vehicle;

generating third and fourth lines which pass through said first and second points respectively, said third and fourth liens being arranged to be tangential to any curve that exists in the selected one of said left and right hand lane edges; and

measuring first and second angles said first angle being defined between said first and third lines and said second angle being defined between said second and fourth lines.

7. A method as claimed in claim 6 further comprising the steps of:

comparing said first angle, said second angle and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

8. A method as claimed in claim 7 wherein said comparing step includes assigning polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are defined on the right of said first and second lines are given a second polarity.

9. A method as claimed in claim 1 wherein said step of observing is carried out using first and second cameras and wherein said step of producing an image comprises producing a stereo image.

10. A method as claimed in claim 9 wherein said step of rectifying comprises converting said stereo image into a planar image.

11. In an unmanned vehicle travelling on a lane of a road,

camera means for observing the road;

second means responsive to said camera means for;

producing an image of the rod;

rectifying the image in a manner which depicts a left hand and right hand edge of the lane on which the vehicle is running and which enables data to be read therefrom;

determining any curvature of the road and the direction in which it is curving;

determining the attitude of the vehicle on the road by determining the angle between the longitudinal axis and an imaginary line which runs along the middle of the lane depicted in the rectified image;

determining the vehicle speed;

determining a variable which varies as a function of road curvature, the distance of the enter of the vehicle from a side of the lane on which the vehicle is running and vehicle speed;

performing a table look-up on a selected map using the attitude angle and the variable, to derive a steering angle; and

means or controlling the range of the steering wheels of the vehicle in accordance with the output of the second means with respect to the longitudinal axis of the vehicle to the derived steering angle.

12. A vehicle claimed in claim 11 wherein said second means

establishes a measurement level line on the rectified image at a location representative of a predetermined distance ahead of said vehicle;

determines first and second points oat the locations where said level lines intersect the left and right hand edges of the lane, respectively;

determines the distance from the point that the longitudinal axis so the vehicle intersects said level line to the left hand edge of the lane;

determines the distance form the point that the longitudinal axis of the vehicle intersects said levelline to the left hand edge of the lane;

generates first and second lines which pass through id said point and which extend parallel with the longitudinal axis of the vehicle and normal to said level;

generates third and fourth lines which pass through said first and second points and which extend tangentially with respect to any curvature which exits in the left and the road lane edges at said first and second points, respectively; and

measures first and second angles, said first angle being defined between said first and third lines and said second angle being defined between said second and fourth lines.

13. A vehicle as claimed in claim 12 wherein said second means further:

compares said first angle, said second angle and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

14. A vehicle as claimed in claim 13 wherein said second means further:

assignees polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are defined on the right of said first and second lines are given a second polarity.

15. A vehicle as claimed in claim 14 wherein second means further:

indicates the first and second second angle data as being erroneous in the event that the polarity of said first and second angles are different.

16. A vehicle as claimed in claim 11 wherein said second means;

establishes first and second level lines on the rectified image, said first and second level lines being drawn on said image at first and second distances ahead of said vehicle, said second distance being greater than the first one;

determines first and second points on a selected one of the left and right hand lane edges, where said first and second level lines intersect the same;

generates first and second lines which pass through said first and second points and which are normal to said first and second level lines respectively and parallel with the longitudinal axis of said vehicle;

generates third and fourth lines which pass through said first and second points respectively, said third and fourth lines being arranged to be tangential to any curve that exists in the selected one of said left and right hand lane edges; and

measures first and second angles, said first angle being defined between said first and third lines and said second angle being defined between said second and fourth lines.

17. A vehicle as claimed in claim 16 wherein said second means further;

compares said first angle, said second angle and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

18. A vehicle as claimed in claim 17 wherein said said second means assigns polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are defined on the right and said first and second lines are given a second polarity.

19. A vehicle as claimed in claim 11 wherein said first means comprises first and second cameras which are arranged to produce a stereo image.

20. A vehicle s claimed in claim 19 wherein said second mean converts said stereo image into a planar image.

21. In an unmanned vehicle

a system for controlling said vehicle, comprising:

first and second cameras for observing the road on which the vehicle is running;

a wheel speed sensor arrangement, said wheel speed sensor arrangement producing a signal indicative of the rotational speed of first and second road wheels and from which the vehicle speed can be derived;

a time computing section operatively connected to said first and second cameras, said image computing section including means for;

producing an image of the road;

rectifying the image in a manner which depicts left and right hand edges of the land on which the vehicle is running; and

determining the curvature of the road and the directions in which it is curving;

a steering control control section operatively connected with said image computing section, said steering control section including means for;

determining the attitude of the vehicle on the road by determining the angle between the longitudinal axis and an imaginary line which runs along the middle of the lane depicted in the rectified image;

determining a variable which carries as a function of road curvature, the distance of the center of the vehicle from a side of the lane on which the vehicle is running and the vehicle speed; and

using the attitude angle and the available to derive a steering angle;

a steering control actuator, said steering control actuator being operatively connected with the steerable wheels of the vehicle;

an actuator control section;

said actuator control section being operatively interposed between said steering control section and said steering control actuator;

said actuator control section controlling the angle of the sterring wheels of the vehicle with respect to the longitudinal axis of the vehicle to the derived steering angle.

22. In an unmanned vehicle, a system for controlling said vehicle, said system comprising:

electronic stereo image data means for depicting the lane in which the vehicle is running;

rectifying means or converting the stereo image data into a planer image on which one or more level lines can be generated, these generated lines intersecting one or both of the edges of the lane in which the vehicle is running;

angle measuring means for measuring the angles between lines which extend from these points of intersection in a direction normal to the level line or liens and which extend tangentially to any curvature which occurs in the road edges at the point of intersection; and

means for recording these angles and computing the difference therebetween, said difference therebetween indicating a best steering angel of the vehicle in accordance with a look-up performed on a selected map using an attitude angle and a variable.

23. A method of controlling a vehicle traveling along a road, comprising the steps of:

producing an image of the road by means of a camera;

rectifying the image into a planar two dimensional image and a three dimensional image of the lane from images which the curvature of the lane, the width of the lane, the distance of the center of the vehicle form a side of the lane, and the attitude of the vehicle on the lane, can be determined, said step of determining the attitude of the vehicle on the lane being achieved by drawing a line along the middle of the lane depicted n the rectified image and determining the angle between the longitudinal axis of the vehicle and the line;

determining vehicle speed by reading the output of a vehicle speed sensor;

deriving a control parameter value which varies with the product of (a) the difference between the width of the lane and the distance of ht center of the vehicle from a side of the lane, and (b) the curvature of the lane, the product being divided by the vehicle speed;

selecting for the control value and attitude one of a plurality of pre-recorded maps from which the most appropriate steering angle can be looked up;

looking-up a steering angle for the selected map; and

steering the vehicle in accordance with the steering angle so derived.

24. A control system for a vehicle, comprising:

a camera for producing an image of the road on which the vehicle is running;

means for rectifying the image into a planar two dimensional image and a three dimensional image of the lane;

means for extracting data, including the curvature of the lane, the width of the lane, the distance of the center of the vehicle from a side of the lane, and the attitude of the vehicle on the lane, from the rectified image, said data extracting means determining the attitude of the vehicle in the lane by drawing a line along the middle of the lane depicted in the rectified image and determining the angel between the longitudinal axis of the vehicle and the line;

means for determining vehicle speed by reading the output of a vehicle speed sensor;

means for deriving a control value which varies with the product of (a) the difference between the width of the lane and the distance of the center of the vehicle from a side of the lane, and (b) the curvature of the lane, the product being divided by the vehicle speed;

means for applying the control value and attitude of the vehicle on the lane to select one a plurality of pre-recorded maps from which the most appropriate steering angel can be looked up; and

means for looking up a steering angel from the selected map and steering the vehicle in accordance with the steering angel derived by look-up.

25. In a method of controlling a vehicle, the steps of:

observing the road;

producing an image of the road;

rectifying the image in a manner which depicts the left and right a hand edges of the lane on which the vehicle is running and which enables data to be read therefrom;

determining the curvature of the road and the direction in which it is curving;

determining the attitude of the vehicle on the road by determining the angle between the longitudinal axis and an imaginary line which runs along the middle of the lane depicted in the rectified image;

determining the vehicle speed;

determining a variable which varies as a function of the road curvature, the distance of the center of the vehicle from a side of the lane on which the vehicle is running and the vehicle speed;

using the attitude angle and the variable to derive a steering angle; and

controlling the angle of the steering wheels of the vehicle with respect to the longitudinal axis of the vehicle to the derived steering angle;

wherein said step of curvature determination comprises;

establishing a measurement level line on the rectified image at a location representative of a predetermined distance ahead of said vehicle;

determining first and second points at the locations where said level lines intersect the left and right hand edges of the lane, respectively;

determining the distance form the point that the longitudinal axis of the vehicle intersects said level line to the left hand edge of the lane;

determining the distance from the point that the longitudinal axis of the vehicle intersects said level line of the left hand edge of the lane;

generating first and second lines which pass through said point and which extend parallel to the longitudinal axis of the vehicle and normal to said level; and

generating third and fourth lines which pass through said first and second points and which extend tangentially with respect to any curvature which exists in the left and the road lane edges at said first and second points, repressively; and

measuring first and second angles, said first angle being defined between said first and third lines and said second angle being defined between said second and fourth lines.

26. A method as claimed in claim 25, further comprising the steps of:

comparing said first angle, said second angle and the difference between said first and second angles in a manner to determine the attitude of the vehicle on said lane.

27. A method as claimed in claim 26, wherein said comparing step includes assigning polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are define don the right of said first and second lines are given a second polarity.

28. A method as claimed din claim 27, wherein the first and second angle data is deemed erroneous in the event that the polarities of said first and second angles are different.

29. A method as claimed in claim 25 wherein said step of curvature determination comprises:

establishing first and second level lines on the rectified image, said first and second level lines being drawn on said image at first and second distances ahead of said vehicle, said second distance being greater than he first distance;

determining first and second points on a selected one of the left and right hand lane edges, where said first and second level lines intersect the same;

generating first and second lines which pass through said first and second points and which are normal to said first and second level lines respectively and parallel with the longitudinal axis of said vehicle;

generating third and fourth lines which pass through said first and second points respectively, said third and fourth lines being arranged to be tangential to any cure that exists in the selected one of said left and right hand lane edges; and

measuring first and second angles, said first angle being defined between said first and third lines and said second angle being defined between said second and fourth lines.

30. A method as claimed din claim 29 further comprising the steps of:

comparing said first angle, said second angel and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

31. A method as claimed din claim 30 wherein said comparing step includes assigning polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said firs and second lines are given a first polarity and angles which are defined on the right of said first and second lines are given a second polarity.

32. In a vehicle travelling in the lane of a road,

first means for observing the road;

second means responsive to said observing means for;

producing an image of the rod;

rectifying the image in a manner which depicts a left hand and right hand edge of the lane on which the vehicle is running and which enables data to be read therefrom;

determining any curvature of the road and the direction in which it is curving;

determining the attitude of the vehicle on the road by determining the angle between the longitudinal axis and an imaginary line which runs along the middle of the lane depicted in the rectified image;

determining the vehicle speed;

determining a variable which aries as a function of road curvature, the distance of the center of the vehicle from a side of the lane on which the vehicle is running and vehicle speed; and

using the attitude angle and the variable to derive a steering angle; and

controlling the angle of the steering wheels of the vehicle in accordance with the output of the second means with respect to the longitudinal axis of the vehicle to the derived steering angle;

wherein;

said second means establishes a measurement level line of the rectified image at a location representative of a predetermined distance ahead of said vehicle;

determines first and second points at the locations where said level lines intersect the left and right hand edges of the lane, respectively;

determines the distance from the point that the longitudinal axis of the vehicle intersects said level line to the left hand edge of the lane;

determines the distance form the point that the longitudinal axis of the vehicle intersects said level line to the left hand edge of the lane;

generates first and second lines which pass through said point and which extend parallel with the longitudinal axis of the vehicle and normal to said level;

generates third and fourth lines which pass through said first and second points and which extend tangentially with respect to any curvature which exits in the left and the road lane edges at said first and second points, respectively; and

measures first and second angels, said first angle being defined between said firs and third lines and said second angle being defined between said second and fourth lines.

33. A vehicle as claimed in claim 32 wherein said second means further assigns polarities to said first and second angels in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are defined on the right of said first and second lines are given a second polarity.

34. A vehicle as claimed in claim 33 wherein second means further indicates the first and second angle data as being erroneous in the event that the polarity of said first and second angels are different.

35. A vehicle as claimed din claim 32, wherein said second means further compares said first angle, said second angle and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

36. A vehicle as claimed din claim 35 wherein said second means establishes first and second level lines on eh rectified image, said first and second level lines being drawn on said image at first and second distances ahead of said vehicle, said second distance being greater than the first one;

determines first and second points on a selected one of the left and right hand lane edges, where said first and second level lines intersect the same;

generates first and second lines which pass through said first and second points and which are normal to said first and second level lines respectively and parallel with the longitudinal axis of said vehicle;

generates third and fourth lines which pass through said first and second points respectively, said third and fourth lines being arranged to be tangential to any curve that exists in the selected one of said left and right hand lane edges; and

measures first and second angles, said first angle being defined between said first and third lines and said second angle being defined between said second and fourth lines.

37. A vehicle as claimed din claim 36 wherein said second means further;

compares said first angle, said scond angle and the difference between said first and second angles in manner to determine the attitude of the vehicle on said lane.

38. A vehicle as claimed din claim 37, wherein said second means assigns polarities to said first and second angles in accordance with a convention wherein angles defined on the left of said first and second lines are given a first polarity and angles which are defined on the right of said first and second lines are given a second polarity.