DRIVING ASSISTANCE APPARATUS

• US 20200130578A1
• Filed: 10/25/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Aisin Corp.
• Original Assignee: Aisin Seiki Co., Ltd. (Aisin Corp.)
• Inventors: MURAKAMI, Shinichiro
• Granted Patent: US 11,440,473 B2
• Time in Patent Office: Days
• CPC Class Codes: B60K 2360/149 Instrument input by detecting viewing direction not otherwise provided for
• CPC Class Codes: B60K 2360/178 Warnings
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/741 Instruments adapted for user detection
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/285 for improving awareness by directing driver's gaze direction or eye points
• CPC Class Codes: B60K 35/65 Instruments specially adapted for specific vehicle types or users, e.g. for left- or right-hand drive
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2011/0003 inside the vehicle
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A driving assistance apparatus includes a gaze detection portion detecting a gaze distribution of a driver for a vehicle, an image acquisition portion acquiring a captured image from an imaging device that captures an image in surroundings of the vehicle, a driver information acquisition portion acquiring driver information that allows identification of the driver for the vehicle, a generation portion generating a personalized saliency map based on the captured image and the driver information, the personalized saliency map that serves as a saliency map for the captured image and that differs depending on the driver, and a determination portion determining whether or not the driver looks at a visual confirmation target in the surroundings of the vehicle by comparing the gaze distribution detected by the gaze detection portion and the personalized saliency map generated by the generation portion.

Claims

1. A driving assistance apparatus comprising:

a gaze detection portion detecting a gaze distribution of a driver for a vehicle;

an image acquisition portion acquiring a captured image from an imaging device that captures an image in surroundings of the vehicle;

a driver information acquisition portion acquiring driver information that allows identification of the driver for the vehicle;

a generation portion generating a personalized saliency map based on the captured image and the driver information, the personalized saliency map that serves as a saliency map for the captured image and that differs depending on the driver; and

a determination portion determining whether or not the driver looks at a visual confirmation target in the surroundings of the vehicle by comparing the gaze distribution detected by the gaze detection portion and the personalized saliency map generated by the generation portion.

2. The driving assistance apparatus according to claim 1, wherein the generation portion learns a generative model by a generative deep neural network, the generative model being configured to generate the personalized saliency map based on the captured image and the driver information,

the generation portion generates the personalized saliency map by the generative model.

3. The driving assistance apparatus according to claim 1, further comprising an alert portion alerting the driver to danger in a case where it is determined that the driver fails to look at the visual confirmation target.

4. The driving assistance apparatus according to claim 3, wherein the driver information acquisition portion acquires the driver information including driving skill of the driver,

the generation portion generates the personalized saliency map that differs depending on the driving skill of the driver based on the captured image and the driver information, and

the determination portion determines the driving skill of the driver based on the gaze distribution detected by the gaze detection portion and the personalized saliency map generated by the generation portion.

5. The driving assistance apparatus according to claim 4, wherein the alert portion changes a frequency of alerts based on a determination result of the driving skill of the driver determined by the determination portion.

6. The driving assistance apparatus according to claim 4, wherein the alert portion differentiates the frequency of alerts between a case where the personalized saliency map for a first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other and a case where the personalized saliency map for a second driving skill higher than the first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other.

7. The driving assistance apparatus according to claim 5, wherein the alert portion differentiates the frequency of alerts between a case where the personalized saliency map for a first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other and a case where the personalized saliency map for a second driving skill higher than the first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other.

8. The driving assistance apparatus according to claim 3, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

9. The driving assistance apparatus according to claim 4, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

10. The driving assistance apparatus according to claim 5, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

11. The driving assistance apparatus according to claim 6, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

12. The driving assistance apparatus according to claim 8, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.