Motor vehicle warning and control system and method

• US 6,226,389 B1
• Filed: 12/28/1999
• Issued: 05/01/2001
• Est. Priority Date: 08/11/1993
• Status: Expired due to Term

• Current Assignee: Jerome H. Lemelson, Robert Pedersen
• Original Assignee: Jerome H. Lemelson, Robert Pedersen
• Inventors: Pedersen, Robert, Lemelson, Jerome H.
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US09/473,350
• Time in Patent Office: 490 Days
• Field of Search: 382/100, 382/104, 382/106, 382/107, 382/181, 382/190, 382/224, 382/226, 382/228, 382/236, 700/1, 700/2, 700/7, 700/17, 700/19, 700/50, 700/61, 700/63, 700/69, 700/76, 700/82, 700/258, 700/259, 700/262, 700/302, 700/304, 701/1, 701/23, 701/27, 701/28, 701/36, 701/40, 701/41, 701/44, 701/70, 701/93, 701/98, 701/96, 701/116, 701/117, 701/225, 701/300, 701/301, 701/302, 706/45, 706/46, 706/52, 706/61, 706/64, 348/135, 348/142, 348/148, 348/149
• US Class Current: 382/104
• CPC Class Codes: B60T 2201/022 Collision avoidance systems
• CPC Class Codes: B60T 2201/024 Collision mitigation systems
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: B60T 2201/082 using alarm actuation
• CPC Class Codes: B60T 2201/083 using active brake actuation
• CPC Class Codes: B60T 2201/085 using several actuators; Coordination of the lane keeping system with other control systems
• CPC Class Codes: B60T 2201/087 using active steering actuation
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 19/11 wherein the cooperating elements are pseudolites or satellite radio beacon positioning system signal repeaters
• CPC Class Codes: G01S 2013/9316 combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9318 Controlling the steering
• CPC Class Codes: G01S 2013/93185 Controlling the brakes
• CPC Class Codes: G01S 2013/9319 Controlling the accelerator
• CPC Class Codes: G01S 2013/932 using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 2013/9321 Velocity regulation, e.g. cruise control
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/9325 : for inter-vehicle distance regulation, e.g. navigating in platoons
• CPC Class Codes: G01S 5/009 : Transmission of differential positioning data to mobile
• CPC Class Codes: G07C 5/0891 : Video recorder in combination with video camera
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A system and method assist the driver of a motor vehicle in preventing accidents or minimizing the effects of same. In one form, a television camera is mounted on a vehicle and scans the roadway ahead of the vehicle as the vehicle travels. Continuously generated video picture signals output by the camera are electronically processed and analyzed by an image analyzing computer, which generates codes, that serve to identify obstacles. A decision computer mounted in the controlled vehicle receives such code signals along with code signals generated by the speedometer or one or more sensors sensing steering mechanism operation and generates control signals. Such code signals maybe displayed, and as synthetic speech or special sound generating and warning means used to warn the driver of the vehicle of approaching and existing hazards. The system may also use the control signals, particularly through application of fuzzy logic, to control the operation of the brakes and steering mechanism of the vehicle to avoid or lessen the effects of a collision. In a particular form, the decision computer may select the evasive action taken from a number of choices, depending on whether and where the detection device senses other vehicles or obstacles.

Claims

1. A method for controlling the travel of a powered vehicle having manual steering and acceleration controls and supporting a ranging apparatus comprising:

(a) as the powered vehicle travels a roadway, measuring the distance to and relative velocity with respect to the vehicle of a plurality of objects in the vicinity of the vehicle;

(b) calculating, with respect to each of the objects, whether the object and the powered vehicle are on a collision course;

(c) when the calculation reveals that a collision is imminent, (i) selecting one of a plurality of state vectors containing a set of binary digits, some of the digits each indicating only whether there is detected an object in one of a plurality of directions from the vehicle that is on a collision course with respect to the vehicle and other of the digits each indicating only whether or not another object is detected in each of the plurality of directions other than the ones on a collision course with respect to the vehicle, (ii) using the selected state vector to select one of a plurality of sets of fuzzy logic inference rules, and (iii) using the selected set of rules to determine a combination of steering and acceleration that is coordinated to attempt to avoid a collision between the powered vehicle and the detected objects;

(d) wherein the fuzzy logic inference rules in each of the plurality of sets are each dependent on at least (i) the distance and relative velocity with respect to the vehicle of at least one of the detected objects on a collision course with respect to the powered vehicle and (ii) in which of the plurality of directions relative to the powered vehicle other detected objects are located; and

(e) applying the determined combination of steering and acceleration to automatically control the steering and acceleration of the powered vehicle.

2. A method in accordance with claim 1 wherein measuring the distance between the powered vehicle and one of the detected objects includes computer processing video picture signals in a manner to measure the width of the rear end of an automobile in front of the powered vehicle in an image defined by video picture signals acquired by a video camera supported by the powered vehicle.

3. A method in accordance with claim 2 further comprising intelligibly indicating the distance between the powered vehicle and the automobile in front on a continuous basis.

4. A method in accordance with claim 1 wherein applying the determined combination of steering and acceleration to automatically control the powered vehicle comprises controlling a brake to slow the forward travel of the powered vehicle.

5. A method in accordance with claim 1 wherein applying the determined combination of steering and acceleration to automatically control the powered vehicle comprises controlling an accelerator to alter the forward travel of the powered vehicle while also altering the direction of travel of the vehicle.

6. A method in accordance with claim 1 further comprising intelligibly indicating when a collision is imminent between one of the objects and the powered vehicle by operating a warning device selected from a group including a warning light, a flashing light, a display, a sound generator, and a speech generator.

7. A method in accordance with claim 6 wherein intelligibly indicating comprises displaying a visually perceptible symbol on a windshield of the powered vehicle.

8. A method in accordance with claim 6 wherein intelligibly indicating comprises generating sounds of select speech recommending a corrective action to take to avoid a collision.

9. A method in accordance with claim 1 further comprising visually indicating the positions of any detected obstacles relative to the powered vehicle.

10. A method in accordance with claim 1 further comprising electro-optically scanning the roadway to detect the objects.

11. A method in accordance with claim 10 further comprising scanning ahead of and in additional directions from the powered vehicle with a plurality of video scanners supported by the powered vehicle and directed in different directions.

12. A method in accordance with claim 11 wherein at least some of the scanners are positioned to scan the roadway in front of the powered vehicle and the roadway behind the powered vehicle.

13. A method in accordance with claim 11 wherein at least some of the scanners are positioned to scan areas in front of the powered vehicle and on both sides of the powered vehicle.

14. A method in accordance with claim 1 wherein part (c)(i) includes ranking each such object calculated as being on a collision course with respect to the vehicle in ascending order of calculated time to collision, and selecting a state vector based on the direction from the vehicle of the highest-ranking object and whether or not another object is detected in each other of the plurality of directions.

15. A method in accordance with claim 1 further comprising measuring the distance and relative velocity between the powered vehicle and all of the objects.

16. A method in accordance with claim 1 further comprising reproducing the selected set of fuzzy logic inference rules from an associative memory.

17. A method in accordance with claim 1 further comprising, upon receipt of a human-issued override command, ceasing the automatic control of the acceleration and steering of the powered vehicle.

18. A method for enhancing safety of a moving vehicle having a steering system and an acceleration system and supporting ranging apparatus capable of producing ranging signals identifying the distance and relative motion between the moving vehicle and selected types of objects in a plurality of directions therefrom, comprising:

(a) using ranging signals identifying the distance and relative motion between the moving vehicle and selected types of objects in a plurality of directions therefrom to determine which, if any, of the objects detected adjacent to the moving vehicle are collision hazards because the object and the moving vehicle are on a collision course;

(b) selecting one of a plurality of state vectors containing a set of binary digits, (i) some of the digits each indicating only whether or not a collision hazard has been determined to be located in one of a plurality of directions from the moving vehicle, and (ii) other of the digits each indicating only whether or not other of the selected objects are detected in each of said plurality of directions other than those directions in which collision hazards are detected;

(c) using the selected state vector to select one of a plurality of rule sets, (i) each rule set comprising a plurality of fuzzy logic vehicle control rules, (ii) each rule specifying an output that depends on an input comprising the calculated distance and motion of at least one of the detected collision hazards relative to the moving vehicle, and (iii) the output of each rule comprising a coordinated combination of changes to the moving vehicle'"'"'s steering and acceleration predetermined to be appropriate for the situation defined by at least the state vector and the rule input;

(d) using the selected set of fuzzy logic control rules to generate output signals indicative of steering and acceleration changes; and

(e) applying the output signals to automatically provide coordinated control of the steering system and the acceleration system to attempt to avoid a collision between the moving vehicle and the collision hazards while also attempting to avoid a collision between the moving vehicle and any other detected objects.

19. A method in accordance with claim 18 wherein some of the fuzzy logic control rules result in output signals that, when applied, cause a change in both the speed and the direction of travel of the vehicle simultaneously.

20. A method in accordance with claim 18 further comprising applying the output signals to intelligibly indicating a warning to a human operator within the moving vehicle.

21. A method in accordance with claim 20 wherein warning the human operator occurs before the acts of part (e), and further comprising performing the acts of part (e) only if the human operator fails to respond to the warning by altering the motion of the vehicle sufficiently to prevent a collision with one of the detected objects from remaining imminent.

22. A method in accordance with claim 21 wherein warning the human operator comprises intelligibly indicating the combination of changes to the moving vehicle'"'"'s steering and acceleration defined by the output signals.

23. A method in accordance with claim 18 further comprising, if more than one collision hazard has been identified:

(a) using the distance and relative motion information for each of the identified collision hazards to priority rank the collision hazards based on expected time to collision; and

(b) reclassifying as objects all of the collision hazards other than the collision hazard with the highest ranking priority;

(c) wherein selecting one of a plurality of state vectors comprises selecting a state vector based on the direction from the vehicle of the collision hazards referenced in part (b)(i) and whether or not another object is detected in each other of the plurality of directions.

24. A method in accordance with claim 23 further comprising using a neural network trained to recognize object types to classify each identified collision hazard into one of a plurality of object types, and wherein priority ranking the collision hazards depends at least in part on the classification.

25. A method in accordance with claim 18 further comprising classifying each detected object into one of a plurality of object types by comparing an image of each object with a plurality of reference images.

26. A method in accordance with claim 18 further comprising using road information, in addition to the state vector, to reproduce from a memory one of the plurality of sets of fuzzy logic vehicle control rules.

27. A method in accordance with claim 18 wherein the plurality of directions consists of directions to the front, to the back, and to both sides of the moving vehicle.

28. A method in accordance with claim 18 wherein the plurality of directions consists of directions to the front and to the back of the moving vehicle.

29. A method in accordance with claim 18 wherein the plurality of directions consists of directions to the front and to two sides of the moving vehicle.

30. A system for operating and controlling a motor vehicle having a power-drive system and controls including an accelerator, a brake, and a steering system comprising:

(a) a ranging device supported by the vehicle, directed toward the front of the vehicle, and structured to generate first signals indicating the distance to and relative motion of some objects in front of the vehicle;

(b) a detector supported by the vehicle, directed away from the vehicle in a direction other than the front of the vehicle, and structured to generate a second signal identifying only whether there is an object in said other direction; and

(c) a fuzzy logic-based computer having a memory containing a plurality of sets of fuzzy inference vehicle control rules, each rule defining a coordinated combination of changes to the vehicle'"'"'s steering and acceleration;

(d) wherein the computer is coupled to the ranging device and the detector and structured to use the first and second signals to select and reproduce from the memory a selected set of less than all of the rules and to apply the selected set of rules to derive command signals with fuzzy logic;

(e) wherein the computer is electrically coupled to the controls;

(f) whereby the command signals are applied to control the accelerator, brake, and steering system of the vehicle in a coordinated way to attempt to avoid collisions between the vehicle and objects in its path of travel without taking evasive action that would cause a collision with objects detected in said other direction.

31. A system in accordance with claim 30 further comprising a visual display inside the vehicle coupled to the ranging device and driven by the first signals to generate symbols representative of objects in the path of the vehicle.

32. A system in accordance with claim 31 wherein the visual display comprises a heads-up display aimed to project images of intelligible information on a front windshield of the vehicle.

33. A system in accordance with claim 30 further comprising a synthetic speech generating system coupled to the computer and driven by the command signals to generate sounds of select words of speech.

34. A system in accordance with claim 30 wherein the first computer is structured to identify one of the objects by comparing the shape of part of the object to a set of standard shapes and generating a second code signal indicating a match, and wherein the second code signal and a measurement of the object from the first signals is used to determine the distance between the vehicle and the object.

35. A system in accordance with claim 30 further comprising a warning device coupled to the ranging device and driven by the first signals to generate a warning signal perceptible to a human when one of the objects is in the path of the vehicle, and wherein the computer is timed to control the operation of the vehicle only if the controls of the vehicle are not first altered sufficiently to avoid a collision with the object in response to indication by the warning device that an obstacle is in the path of travel of the vehicle.

36. A system in accordance with claim 30 wherein the ranging device comprises an image-generating camera.

37. A system in accordance with claim 30 wherein the ranging device comprises a radar-based ranging system.

38. A system in accordance with claim 30:

(a) further comprising a second scanning device supported by the vehicle, directed away from the vehicle in a direction other than the front of the vehicle, and structured to generate second signals modulated with information relating to objects in the field of view of the second scanning device; and

(b) further comprising a third computer coupled to the second scanning device and structured to analyze the second signals as the vehicle travels and to produce second code signals on a third output, which second code signals are indicative of distances and relative motion between the vehicle and each of the objects in the field of view of the second scanning device; and

(c) wherein the second computer is also coupled to the third output and structured to use the first and second code signals to reproduce from the memory a selected set of the fuzzy inference vehicle control rules that are applied using fuzzy logic to control operation of the vehicle to attempt to avoid collisions both with objects in the path of travel of the vehicle and objects in the field of view of the second scanning device.

39. A system in accordance with claim 38:

(a) further comprising a third scanning system supported by the vehicle, directed away from the vehicle in directions other than the front of the vehicle and other than the direction of the second scanning device, and structured to generate third signals modulated with information relating to objects not in the field of view of either the first or second scanning devices;

(b) further comprising a fourth computer coupled to the third scanning system and structured to analyze the third signals as the vehicle travels and to produce third code signals on a fourth output, which third code signals are indicative of distances and relative motion between the vehicle and each of the objects in the field of view of the third scanning system;

(c) wherein the first and second scanning devices and the third scanning system together cover substantially all of an area surrounding the vehicle; and

(d) wherein the second computer is also coupled to the fourth output and structured to use the first, second, and third code signals to reproduce from the memory a selected set of the fuzzy inference vehicle control rules that are applied using fuzzy logic to control operation of the vehicle to attempt to avoid collisions both with objects in the path of travel of the vehicle and objects in the fields of view of the second scanning device and the third scanning system.

40. A system in accordance with claim 30 further comprising an override controller coupled to the computer so as, when activated by a driver, to prevent the command signals from controlling the motor vehicle.