Driver assistance system with vehicle states, environment and driver intention

• US 7,974,748 B2
• Filed: 08/16/2006
• Issued: 07/05/2011
• Est. Priority Date: 08/18/2005
• Status: Active Grant

• Current Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Original Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Inventors: Schneider, Martin, Gayko, Jens, Goerick, Christian
• Primary Examiner(s): Black; Thomas G
• Assistant Examiner(s): Kong; Sze-Hon
• Application Number:
  US11/505,541
• Publication Number:
  US 20070043491A1
• Time in Patent Office: 1,784 Days
• Field of Search: 701/41, 701/300, 701/28, 701/301, 340/426.22, 340/426.23, 340/436
• US Class Current: 701/28
• CPC Class Codes: B60K 28/165 acting on elements of the vehicle drive train other than the propulsion unit and brakes, e.g. transmission, clutch, differential acting on brakes B60T8/17
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A driver assistance system with an architecture of general use for applications related to driver assistance, passenger safety and comfort and communication with external systems, e.g. for diagnosis and maintenance purposes, comprises (in one embodiment): a first interface for sensing external signals coming from the environment of a vehicle, a second interface for sensing internal signals of the vehicle, a first subsystem for fusing the output signals of the first interface and the second interface in order to generate a representation of the vehicle'"'"'s environment, a second subsystem for modeling the current state of the vehicle being provided with the output signals of the first subsystem, a third subsystem for modeling the behavior of a driver, a fourth subsystem for interpreting the current scene represented by the first subsystem in order to determine characteristic states of the current external situation of the vehicle, and a fifth subsystem that generates stimuli for the actuators of the vehicle and/or the driver based on the output signals of the second, the third and the fourth subsystem.

Claims

1. A driver assistance system comprising:

a first interface for sensing external signals received from an environment of a vehicle and generating a first interface output signal, the first interface comprising projective sensors and non-projective sensors,

a second interface for sensing internal signals of the vehicle and generating a second interface output signal,

a first subsystem comprising

an information fusion module, a vehicle environment representation module, a vehicle environment interpretation module, and a knowledge base for storing representations of detectable object associated attributes,

wherein the information fusion module identifies an object based on the first interface output signal and provides the identity of the object to the knowledge base,

wherein the knowledge base receives said identity of the object and provides attribute signals representing at least one object attribute of the identified object to the vehicle environment representation module, and

wherein the vehicle environment representation module generates a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute,

wherein the vehicle environment interpretation module sends data to the vehicle environment representation module to guide a filtering process that filters identified objects from the central model of the vehicle environment based on relevance,

a second subsystem for modeling a current internal state of the vehicle based on the second interface output signal,

a third subsystem for modeling a driver intention,

a fourth subsystem for analyzing the central model of the vehicle environment represented by the first subsystem in order to distinguish external environmental conditions relevant to the vehicle, and

a fifth subsystem for ranking a hierarchy of action signals, the action signals generated based on the output signals of the second, the third and the fourth subsystems, wherein the ranking is based on the modeled driver intention, and for generating stimuli for an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

2. The system according to claim 1,

wherein the fifth subsystem is designed to additionally use additional traffic rules depending on the central model of the vehicle environment.

3. The system according to claim 1,

wherein the fifth subsystem is adapted to feed-back signals to the fourth subsystem representing information related to the generated stimuli and

the fourth subsystem is adapted to analyzes the central model of the vehicle environment in response to the feed-back signal of the fifth subsystem.

4. The system according to claim 1, further comprising

an urban/rural detection subsystem for providing a description of a global scenario, adapted to determine the global scenario by an output signal of the fourth subsystem and to output a signal representing the determined global scenario, wherein the subsystem for generating stimuli for the actuators of the vehicle and/or the driver is adapted to generate these stimuli in response to the output signal of the urban/rural detection subsystem.

5. The system of claim 4, wherein the urban/rural detection subsystem provides an output signal for traffic rule selection.

6. The system according to claim 1, wherein the fourth subsystem is adapted to further generate a stimulus for the third subsystem and the third subsystem is adapted to model the behavior of the driver in response to the input signal of the fourth subsystem and to an input signal from a human machine interface representing an action of the driver.

7. The system according to claim 1, wherein the subsystem for generating stimuli for the actuators and/or the driver is further adapted to generate these stimuli in response to input signals from passenger monitoring.

8. The system according to claim 1, wherein each representation of an object further comprises an object categorization for use in action selection in the fifth subsystem.

9. The system according to claim 1, for use in a vehicle.

10. The system according to claim 1, wherein the central model of the vehicle environment may further comprise weather, road, or traffic information.

11. The system of claim 1, wherein the current task is a lane keeping, and the filtering process further comprises retaining only those identified objects which are in a nearest lane with respect to the vehicle.

12. The system of claim 1, wherein one of the at least one object attributes of the identified object is a rigidity of the object.

13. A driver assistance system, the system comprising:

a first interface for sensing external signals coming from an environment of a vehicle and generating a first interface output signal, the first interface comprising projective sensors and non-projective sensors,

a second interface for sensing internal signals of the vehicle and generating a second interface output signal,

a first subsystem comprising

an information fusion module, a vehicle environment representation module, a vehicle environment interpretation module, and a knowledge base for storing representations of detectable object associated attributes,

wherein the information fusion module identifies an object based on the first interface output signals and provides the identity of the object to the knowledge base,

wherein the knowledge base receives said identity of the object and provides attribute signals representing at least one attribute of the identified object to the vehicle environment representation module, and

wherein the vehicle environment representation module generates a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute,

wherein the vehicle environment interpretation module sends data to the vehicle environment representation module to guide a filtering process that filters identified objects from the central model of the vehicle environment based on relevance,

a second subsystem for modeling a current internal state of the vehicle based on the second interface output signal,

a third subsystem for modeling a driver intention,

a fourth subsystem for analyzing the central model of the vehicle environment represented by the first subsystem, for ranking a hierarchy of action signals, the action signals generated based on the output signals of the second and the third subsystems, wherein the ranking is based on the modeled driver intention, and for generating stimuli for an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

14. The system according to claim 13, wherein the fourth subsystem is designed to use additional traffic rules depending on the central model of the vehicle environment.

15. The system according to claim 13, further comprising

an urban/rural detection subsystem for providing a description of a global scenario, adapted to determine the global scenario by an output signal of the fourth subsystem and to output a signal representing the determined global scenario, wherein the subsystem for generating stimuli for the actuators of the vehicle and/or the driver is adapted to generate these stimuli in response to the output signal of the urban/rural detection subsystem.

16. The system of claim 15, wherein the urban/rural detection subsystem provides an output signal for traffic rule selection.

17. The system according to claim 13, wherein the fourth subsystem is adapted to further generate a stimulus for the third subsystem and the third subsystem is adapted to model the behavior of the driver in response to the input signal of the fourth subsystem and to an input signal from a human machine interface representing an action of the driver.

18. The system according to claim 13, wherein the subsystem for generating stimuli is further adapted to generate these stimuli in response to input signals from passenger monitoring.

19. The system according to claim 13, wherein each representation of an object further comprises an object categorization for use in action selection in the fourth subsystem.

20. A system according to claim 13, for use in a vehicle.

21. The system according to claim 13, wherein the central model of the vehicle environment may further comprise weather, road, or traffic information.

22. A method for assisting a driver of a vehicle, comprising the following steps:

sensing external signals of the vehicle using projective sensors and non-projective sensors,

sensing internal signals of the vehicle,

identifying an object based on the external signals of the vehicle and providing the identity of the object to a knowledge base for storing representations of detectable object associated attributes, which in response provides attribute signals representing at least one attribute of the identified object,

generating a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute,

filtering identified objects from the central model of the vehicle environment based relevance,

modeling a driver intention,

analyzing the central model of the vehicle environment in order to distinguish external environmental conditions relevant to the vehicle, and

combining the analyzed central model of the vehicle environment and the behavior of the driver in order to generate rank a hierarchy of action signals, the action signals based on the relevant identified objects, wherein the ranking is based on the modeled driver intention, and in order to output a stimulus for an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

23. A computer software program product embodied on a computer readable medium which when executed by a computer performs the method steps of:

sensing external signals of a vehicle using projective sensors and non-projective sensors,

sensing internal signals of the vehicle,

identifying an object based on the external signals of the vehicle and providing the identity of the object to a knowledge base for storing representations of detectable object associated attributes, which in response provides attribute signals representing at least one attribute of the identified object,

generating a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute,

filtering identified objects from the central model of the vehicle environment based on relevance,

modeling a driver intention,

analyzing the central model of the vehicle environment in order to distinguish external environmental conditions relevant to the vehicle, and

combining the analyzed central model of the vehicle environment and the behavior of the driver in order to generate rank a hierarchy of action signals, the action signals based on the relevant identified objects, wherein the ranking is based on the modeled driver intention, and in order to output a stimulus for an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

24. The computer program product according to claim 23 comprising firmware, adapted for flashing the firmware into a programmable device of a vehicle.

25. A data storage device onto which the computer software program product of claim 23 is stored.

26. A method for assisting a driver of a vehicle, comprising the following steps:

sensing external signals of the vehicle using projective sensors and non-projective sensors,

sensing internal signals of the vehicle,

identifying an object based on the external signals of the vehicle and providing the identity of the object to a knowledge base for storing representations of detectable object associated attributes, which in response provides attribute signals representing at least one attribute of the identified object,

generating a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute,

filtering identified objects from the central model of the vehicle environment based on relevance,

modeling a driver intention, and

analyzing the central model of the vehicle environment, ranking a hierarchy of action signals, the action signals based on the relevant identified objects, wherein the ranking is based on the modeled driver intention, and generating a stimulus for at least one of an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

27. The method of claim 26 analyzing the central model of the vehicle environment in order to distinguish external environmental conditions relevant to the vehicle.

28. A computer software program product embodied on a computer readable medium which when executed by a computer performs the method steps of:

sensing external signals of the vehicle using projective sensors and non-projective sensors,

sensing internal signals of the vehicle,

identifying an object based on the external signals of the vehicle and providing the identity of the object to a knowledge base for storing representations of detectable object associated attributes, which in response provides attribute signals representing at least one attribute of the identified object,

generating a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute,

filtering identified objects from the central model of the vehicle environment based on relevance,

modeling a driver intention, and

analyzing the central model of the vehicle environment, ranking a hierarchy of action signals, the action signals based on the relevant identified objects, wherein the ranking is based on the modeled driver intention, and generating a stimulus for at least one of an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

29. The computer program product according to claim 28 comprising firmware, adapted for flashing the firmware into a programmable device of a vehicle.

30. A data storage device onto which the computer software program product of claim 28 is stored.

31. A driver assistance system comprising:

a first interface for sensing external signals received from an environment of a vehicle and generating a first interface output signal,

a second interface for sensing internal signals of the vehicle and generating a second interface output signal,

a first subsystem comprising

an information fusion module, a vehicle environment representation module, and a knowledge base for storing representations of detectable object associated attributes,

wherein the information fusion module identifies an object based on the first interface output signal and provides the identity of the object to the knowledge base,

wherein the knowledge base receives said identity of the object and provides attribute signals representing at least one object attribute of the identified object to the vehicle environment representation module, and

wherein the vehicle environment representation module generates a central model of the vehicle environment comprising at least one representation of an object, wherein each representation of an object comprises the identity of the object and at least one object attribute, and

a second subsystem for modeling a current internal state of the vehicle based on the second interface output signal,

a third subsystem for modeling a driver intention,

a fourth subsystem for analyzing the central model of the vehicle environment represented by the first subsystem in order to distinguish external environmental conditions relevant to the vehicle, and

a fifth subsystem for ranking a hierarchy of action signals, the action signals generated based on the output signals of the second, the third and the fourth subsystems, wherein the ranking is based on the modeled driver intention, and for generating stimuli for an actuator of the vehicle or the driver based on a highest ranked action signal in the hierarchy.

32. The system of claim 31, wherein one of the at least one object attributes of the identified object is a rigidity of the object.