Intelligent driver assist system based on multimodal sensor fusion

• US 8,976,040 B2
• Filed: 02/15/2013
• Issued: 03/10/2015
• Est. Priority Date: 02/16/2012
• Status: Active Grant

• Current Assignee: Bianca Ray Avalani
• Original Assignee: Bianca Ray Avalani
• Inventors: Ray Avalani, Bianca
• Primary Examiner(s): MULLEN, THOMAS J
• Application Number:
  US13/768,979
• Publication Number:
  US 20130222127A1
• Time in Patent Office: 753 Days
• Field of Search: 340/901, 340/903, 340/905, 340/435, 340/439, 348/148, 348/149
• US Class Current: 340/903
• CPC Class Codes: B60Q 1/00 Arrangement of optical signalling or lighting devices, the mounting or supporting thereof or circuits therefor for lighting vehicle interior B60Q3/00
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2520/00 Input parameters relating to overall vehicle dynamics
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

An apparatus and method of providing driver assistance for performing a driving action include obtaining video information adjacent to a vehicle, obtaining audio information adjacent to the vehicle, optionally obtaining movement-related information, and generating a driver assist indication based on a combination of the video information and the audio information, and, optionally, the movement-related information.

Claims

1. A method of providing driver assistance for performing a driving action, comprising:

obtaining video information adjacent to a vehicle;

obtaining audio information adjacent to the vehicle; and

generating, via a decision making processor executing one or more decision algorithms, a driver assist indication based on a combination of the video information and the audio information, the generating comprising;

comparing the video information to a video threshold corresponding to video information for which performing the driving action is acceptable;

comparing the audio information to an audio threshold corresponding to audio information for which performing the driving action is acceptable; and

generating the driver assist indication with a positive value or a negative value based on a result of the comparing.

2. The method of claim 1, further comprising:

obtaining vehicle movement-related information; and

wherein generating the driver assist indication is further based on the vehicle movement-related information.

3. The method of claim 2, wherein generating the driver assist indication further comprises:

comparing the obtained vehicle movement-related information to a movement-related threshold corresponding to movement information for which performing the driving action is acceptable.

4. The method of claim 3, wherein generating the driver assist indication further comprises generating the driver assist indication with the negative value when at least one of the obtained video information, the obtained audio information, or the obtained vehicle movement-related information is less than a respective one of the video threshold, the audio threshold, or the movement threshold.

5. The method of claim 1, wherein:

obtaining the audio information further comprises obtaining the audio information from inside of the vehicle; and

generating the driver assist indication is further based on the audio information obtained from the inside of the vehicle.

6. The method of claim 1, further comprising:

estimating a velocity of an object based on the obtained video information;

canceling noise from the obtained video information to define noise-canceled video information;

extracting features from the noise-canceled video information; and

determining whether the extracted features identify that the object is a vehicle.

7. The method of claim 1, further comprising:

performing time domain analysis of the obtained audio information;

performing frequency domain analysis of the obtained audio information;

analyzing signal intensities in the obtained audio information based on at least one of the time domain analysis and the frequency domain analysis; and

determining whether the signal intensities identify a vehicle.

8. The method of claim 1, further comprising:

receiving notification of an intended driving action; and

wherein obtaining the video information, obtaining the audio information, and generating the driver assist indication is in response to the notification of the intended driving action.

9. A computer program product comprising:

a non-transitory computer-readable medium having control logic stored therein, the control logic comprising;

at least one instruction for causing a computer to obtain video information adjacent to a vehicle;

at least one instruction for causing the computer to obtain audio information adjacent to the vehicle; and

at least one instruction for causing the computer to generate a driver assist indication based on a combination of the video information and the audio information,

wherein the at least one instruction for causing the computer to generate a driver assist indication is further for causing the computer to;

compare the video information to a video threshold corresponding to video information for which performing a driving action is acceptable;

compare the audio information to an audio threshold corresponding to audio information for which performing the driving action is acceptable; and

generate the driver assist indication with a positive value or a negative value based on a result of each comparison.

10. The computer program product of claim 9, wherein the control logic further comprises:

at least one instruction for causing the computer to obtain vehicle movement-related information; and

wherein the at least one instruction for causing the computer to generate the driver assist indication is further based on the vehicle movement-related information.

11. An apparatus to provide driver assistance for performing a driving action, comprising:

a video processor configured to obtain video information adjacent to a vehicle;

an audio processor configured to obtain audio information adjacent to the vehicle; and

a decision making processor to generate a driver assist indication based on a combination of the video information and the audio information, the decision making processor being configured to;

compare the video information to a video threshold corresponding to video information for which performing the driving action is acceptable;

compare the audio information to an audio threshold corresponding to audio information for which performing the driving action is acceptable; and

generate the driver assist indication with a positive value or a negative value based on a result of each comparison.

12. The apparatus of claim 11, further comprising:

a movement-related processor configured to obtain a vehicle movement-related information; and

wherein the decision making processor is further configured to generate the driver assist indication based on the vehicle movement-related information.

13. The apparatus of claim 12, wherein the decision making processor is further configured to:

compare the vehicle movement-related information to a movement-related threshold corresponding to movement information for which performing the driving action is acceptable.

14. The apparatus of claim 13, wherein the decision making processor is further configured to generate the driver assist indication with the negative value when at least one of the obtained video information, the obtained audio information, or the obtained vehicle movement-related information is less than a respective one of the video threshold, the audio threshold, or the movement threshold.

15. The apparatus of claim 11, wherein the audio processor configured to obtain the audio information from inside of the vehicle, wherein the decision making processor is further configured to generate the driver assist indication based on the obtained audio information from the inside of the vehicle.

16. The apparatus of claim 11, wherein the video processor comprises a video processing algorithm configured to:

estimate a velocity of an object based on the obtained video information;

cancel noise from the obtained video information to define noise-canceled video information;

extract features from the noise-canceled video information; and

determine whether the extracted features identify that the object is a vehicle.

17. The apparatus of claim 11, wherein the audio processor comprises an audio processing algorithm configured to:

perform time domain analysis of the obtained audio information;

perform frequency domain analysis of the obtained audio information;

analyze signal intensities in the obtained audio information based on at least one of the time domain analysis and the frequency domain analysis; and

determine whether the signal intensities identify a vehicle.

18. The apparatus of claim 11, wherein the video processor is further configured to obtain the video information, the audio processor is further configured to obtain the audio information, and the decision making processor is further configured to generate the driver assist indication in response to a notification of an intended driving action.