INTELLIGENT DRIVER ASSIST SYSTEM BASED ON MULTIMODAL SENSOR FUSION

• US 20130222127A1
• Filed: 02/15/2013
• Published: 08/29/2013
• Est. Priority Date: 02/16/2012
• Status: Active Grant

• Current Assignee: Bianca Ray Avalani
• Original Assignee: Bianca Ray Avalani
• Inventors: RAY AVALANI, Bianca
• Granted Patent: US 8,976,040 B2
• Time in Patent Office: Days
• US Class Current: 340/436
• CPC Class Codes: B60Q 1/00 Arrangement of optical signalling or lighting devices, the mounting or supporting thereof or circuits therefor for lighting vehicle interior B60Q3/00
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2520/00 Input parameters relating to overall vehicle dynamics
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

An apparatus and method of method of providing driver assistance for performing a driving action include obtaining video information adjacent to a vehicle, obtaining audio information adjacent to the vehicle, optionally obtaining movement-related information, and generating a driver assist indication based on a combination of the video information and the audio information, and, optionally, the movement-related information.

Claims

1. A method of providing driver assistance for performing a driving action, comprising:

obtaining video information adjacent to a vehicle;

obtaining audio information adjacent to the vehicle; and

generating a driver assist indication based on a combination of the video information and the audio information.

2. The method of claim 1, further comprising:

obtaining a vehicle movement-related information; and

wherein generating the driver assist indication is further based on the vehicle movement-related information.

3. The method of claim 1, wherein:

obtaining the audio information further comprises obtaining from inside of the vehicle; and

generating the driver assist indication is further based on the audio information from the inside of the vehicle.

4. The method of claim 1, wherein generating the driving assist indication further comprises:

comparing the video information to a video threshold corresponding to video clearance for performing the driving action;

comparing the audio information to an audio threshold corresponding to audio clearance for performing the driving action; and

generating the driving assist indication with a positive value or a negative value based on a result of the comparing.

5. The method of claim 2, wherein generating the driving assist indication further comprises:

comparing the video information to a video threshold corresponding to video clearance for performing the driving action;

comparing the audio information to an audio threshold corresponding to audio clearance for performing the driving action;

comparing the vehicle movement-related information to a movement-related threshold corresponding to movement clearance for performing the driving action; and

generating the driving assist indication with a positive value or a negative value based on a result of the comparing.

6. The method of claim 5, wherein generating the driving assist indication further comprises generating the driving assist indication with the negative value when at least one of the video information, the audio information, or the vehicle movement-related vehicle information is less than a respective one of the video threshold, the audio threshold, or the movement threshold.

7. The method of claim 1, further comprising:

estimating an object velocity from the video information;

canceling noise from the video information to define noise canceled video information;

extracting features from the noise canceled video information; and

determining whether the extracted features identify a vehicle.

8. The method of claim 1, further comprising:

performing time domain analysis of the audio information;

performing frequency domain analysis of the audio information;

analyzing signal intensities in the audio information based on at least one of the time domain analysis and the frequency domain analysis; and

determining whether the signal intensities identify a vehicle.

9. The method of claim 1, further comprising:

receiving notification of an intended driving action; and

wherein obtaining the video information, obtaining the audio information, and generating the driver assist indication is in response to the notification of the intended driving action.

10. A computer program product, comprising:

a computer-readable medium comprising;

at least one instruction for causing a computer to obtain video information adjacent to a vehicle;

at least one instruction for causing the computer to obtain audio information adjacent to the vehicle; and

at least one instruction for causing the computer to generate a driver assist indication based on a combination of the video information and the audio information.

11. The computer program product of claim 10, further comprising:

at least one instruction for causing the computer to obtain a vehicle movement-related information; and

wherein the at least one instruction for causing the computer to generate the driver assist indication is further based on the vehicle movement-related information.

12. An apparatus to provide driver assistance for performing a driving action, comprising:

a video processor configured to obtain video information adjacent to a vehicle;

an audio processor configured to obtain audio information adjacent to the vehicle; and

a decision making processor to generate a driver assist indication based on a combination of the video information and the audio information.

13. The apparatus of claim 12, further comprising:

a movement-related processor configured to obtain a vehicle movement-related information; and

wherein the decision making processor is further configured to generate the driver assist indication based on the vehicle movement-related information.

14. The apparatus of claim 12, wherein the audio processor configured to obtain the audio information from inside of the vehicle, wherein the decision making processor is further configured to generate the driver assist indication based on the audio information from the inside of the vehicle.

15. The apparatus of claim 12, wherein the decision making processor is further configured to:

compare the video information to a video threshold corresponding to video clearance for performing the driving action;

compare the audio information to an audio threshold corresponding to audio clearance for performing the driving action; and

generate the driving assist indication with a positive value or a negative value based on a result of each comparison.

16. The apparatus of claim 13, wherein the decision making processor is further configured to:

compare the video information to a video threshold corresponding to video clearance for performing the driving action;

compare the audio information to an audio threshold corresponding to audio clearance for performing the driving action;

compare the vehicle movement-related information to a movement-related threshold corresponding to movement clearance for performing the driving action; and

generate the driving assist indication with a positive value or a negative value based on a result of the comparing.

17. The apparatus of claim 16, wherein the decision making processor is further configured to generate the driving assist indication with the negative value when at least one of the video information, the audio information, or the vehicle movement-related vehicle information is less than a respective one of the video threshold, the audio threshold, or the movement threshold.

18. The apparatus of claim 12, wherein the video processor comprises a video processing algorithm configured to:

estimate an object velocity from the video information;

cancel noise from the video information to define noise canceled video information;

extract features from the noise canceled video information; and

determine whether the extracted features identify a vehicle.

19. The apparatus of claim 12, wherein the audio processor comprises an audio processing algorithm configured to:

perform time domain analysis of the audio information;

perform frequency domain analysis of the audio information;

analyze signal intensities in the audio information based on at least one of the time domain analysis and the frequency domain analysis; and

determine whether the signal intensities identify a vehicle.

20. The apparatus of claim 12, wherein the video processor is further configured to obtain the video information, the audio processor is further configured to obtain the audio information, and the decision making processor is further configured to generate the driver assist indication in response to a notification of an intended driving action.