Driving assistance method, and driving assistance device, driving control device, vehicle, driving assistance program, and recording medium using said method

• US 10,252,726 B2
• Filed: 04/15/2016
• Issued: 04/09/2019
• Est. Priority Date: 04/21/2015
• Status: Active Grant

• Current Assignee: Panasonic Automotive Systems Company of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Emura, Koichi, Tsuji, Masanaga, Mori, Toshiya, Nakai, Wataru
• Primary Examiner(s): To, Tuan C
• Application Number:
  US15/565,887
• Publication Number:
  US 20180093676A1
• Time in Patent Office: 1,089 Days
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/167 Vehicle dynamics information
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 2360/179 Distances to obstacles or vehicles
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60R 16/02 electric constitutive elements
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2520/10 Longitudinal speed
• CPC Class Codes: B60W 2540/10 Accelerator pedal position
• CPC Class Codes: B60W 2540/12 Brake pedal position
• CPC Class Codes: B60W 2540/18 Steering angle
• CPC Class Codes: B60W 2540/20 Direction indicator values
• CPC Class Codes: B60W 2540/215 Selection or confirmation of options
• CPC Class Codes: B60W 2540/30 Driving style
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/802 Longitudinal distance
• CPC Class Codes: B60W 2554/804 Relative longitudinal speed
• CPC Class Codes: B60W 2556/10 : Historical data
• CPC Class Codes: B60W 2556/55 : using telemetry
• CPC Class Codes: B60W 2720/10 : Longitudinal speed
• CPC Class Codes: B60W 30/09 : Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/18163 : Lane change; Overtaking manoeuvres
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 50/08 : Interaction between the driver and the control system
• CPC Class Codes: B60W 50/082 : Selecting or switching between different modes of propelling
• CPC Class Codes: B60W 50/085 : Changing the parameters of the control units, e.g. changing limit values, working points by control input
• CPC Class Codes: B60W 50/10 : Interpretation of driver requests or demands
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0027 : using trajectory prediction for other traffic participants
• CPC Class Codes: B60W 60/0053 : from vehicle to occupant
• CPC Class Codes: G01C 21/3484 : Personalized, e.g. from learned user behaviour or user-defined profiles
• CPC Class Codes: G01C 21/3605 : Destination input or retrieval
• CPC Class Codes: G01C 21/3617 : using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06F 3/048 : Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/096888 : where input information is obtained using learning systems, e.g. history databases

Abstract

The driving assistance device acquires, from an autonomous driving controller that determines an action of a vehicle during autonomous driving of the vehicle, action information indicating a first action that the vehicle is caused to execute. The driving assistance device acquires, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result. The driving assistance device determines a second action which is executable in place of the first action, based on the detection information. The driving assistance device generates a first image representing the first action and a second image representing the second action. The driving assistance device outputs the first image and the second image to a notification device such that the first image and the second image are displayed within a fixed field of view of a driver of the vehicle.

Claims

1. A driving assistance device comprising:

an action information input unit, which, in operation, acquires, from an autonomous driving controller that determines an action of a vehicle during autonomous driving of the vehicle, action information indicating a first action that the vehicle is caused to execute;

a detection information input unit, which, in operation, acquires, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result;

a candidate determination unit, which, in operation, determines, based on the detection information, at least one second action that the vehicle is caused to execute after the first action indicated by the action information, the first action and the at least one second action adjusting movement of the vehicle;

an image generator, which, in operation, generates a first image representing the first action indicated by the action information and a second image representing the at least one second action; and

an image output unit, which, in operation, outputs the first image and the second image to a display in the vehicle, wherein

the at least one second action includes a plurality of second actions,

the candidate determination unit, in operation, determines the plurality of second actions and determines a priority order of each of the plurality of second actions,

the image generator, in operation, generates a plurality of third images representing the plurality of second actions,

the image output unit, in operation, outputs the plurality of third images according to priority orders to the display in the vehicle to display the plurality of third images in a manner according to the priority orders.

2. The driving assistance device according to claim 1, further comprising:

an accumulation unit, which, in operation, accumulates statistical information indicating relevance between the surrounding situation and the travel state of the vehicle and the action of the vehicle,

wherein the candidate determination unit, in operation, determines the at least one second action based on the statistical information and the detection information.

3. The driving assistance device according to claim 1, further comprising:

a communication interface, which, in operation, communicates with, through a communication network, an accumulation unit, provided outside of the vehicle, configured to accumulate statistical information indicating relevance between the surrounding situation and the travel state of the vehicle and the action of the vehicle,

wherein the candidate determination unit, in operation, accesses the statistical information in the accumulation unit through the communication interface, and determines the at least one second action based on the statistical information and the detection information.

4. The driving assistance device according to claim 1, wherein

the image generator, in operation, generates an additional image representing a time until the at least one second action is executed, and

the image output unit, in operation, outputs the additional image to the display in the vehicle, and displays the second image including the additional image.

5. The driving assistance device according to claim 1, further comprising:

a command output unit, which, in operation, outputs, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, after the first image and the second image are displayed on the display in the vehicle for a predetermined time.

6. The driving assistance device according to of claim 1, further comprising:

an operation signal input unit, which, in operation, receives an operation instruction for designating the action of the vehicle; and

a command output unit, which, in operation, outputs, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, when the operation instruction is not received within a predetermined time for which the first image and the second image are displayed on the display in the vehicle.

7. The driving assistance device according to claim 1, further comprising:

an operation signal input unit, which, in operation, receives an operation instruction for designating the action of the vehicle; and

a command output unit, which, in operation, outputs, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, when an operation instruction for selecting the second image is received within a predetermined time for which the first image and the second image are displayed on the display in the vehicle.

8. The driving assistance device according to of claim 1, further comprising:

a command output unit,

wherein, in response to an operation for designating the at least one second action being input, the image output unit outputs, to the display, an inquiry image for designating execution or reservation of the at least one second action, and

in response to an operation for designating execution being input while the inquiry image is displayed, the command output unit outputs, to the autonomous driving controller at a first timing, a command to cause the vehicle to execute the at least one second action, and

in response to an operation for designating reservation being input while the inquiry image is displayed, the command output unit outputs the command to the autonomous driving controller at a second timing later than the first timing.

9. The driving assistance device according to claim 8, wherein in response to the operation for designating the reservation being input while the inquiry image is displayed, the image output unit outputs, to the display, a fourth image representing that the at least one second action is currently reserved.

10. The driving assistance device according to claim 5, wherein after the command is output to the autonomous driving controller, the action information input unit acquires first action information indicating a first action which has been updated according to the command.

11. The driving assistance device according to claim 1, further comprising:

an operation input unit, which, in operation, receives an operation instruction for designating an action of the vehicle to be executed during autonomous driving of the vehicle,

wherein, in response to an operation instruction for selecting the second image being received within a predetermined time for which the first image and the second image are displayed on the display in the vehicle, the image generator generates an inquiry image making an inquiry to a driver of the vehicle as to whether to immediately execute or to execute after a predetermined time the at least one second action represented by the second image which has been selected, and

the image output unit, in operation, outputs the inquiry image to the display.

12. The driving assistance device according to claim 11, further comprising:

a command output unit,

wherein, in response to the operation input unit receiving an operation instruction indicating that immediate execution of the at least one second action represented by the second image is selected in the inquiry image, the command output unit transmits, to the autonomous driving controller, a control command to cause the vehicle to immediately execute the at least one second action represented by the second image in place of the first action represented by the first image.

13. The driving assistance device according to claim 11, further comprising:

a command output unit,

wherein, in response to the operation input unit receiving an operation instruction indicating that execution of the at least one second action represented by the second image after a predetermined time is selected in the inquiry image, the command output unit transmits, to the autonomous driving controller, a control command to cause the vehicle to execute the at least one second action represented by the second image after a predetermined time.

14. The driving assistance device according to of claim 1, wherein the image output unit outputs the first image and the second image to the display in the vehicle to display the first image and the second image within a fixed field of view of the driver of the vehicle.

15. The driving assistance device according to claim 1 wherein the image output unit outputs the first image and the second image to the display in the vehicle at the same time.

16. A driving control device comprising:

an autonomous driving controller, which, in operation, determines an action of a vehicle during autonomous driving of the vehicle;

a detection information input unit, which, in operation, acquires, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result;

a candidate determination unit, which, in operation, determines, based on the detection information, at least one second action that the vehicle can be caused to execute after a first action that the autonomous driving controller causes the vehicle to execute, the first action and the at least one second action adjusting movement of the vehicle;

an image generator, which, in operation, generates a first image representing the first action and a second image representing the second action; and

an image output unit, which, in operation, outputs the first image and the second image to a display in the vehicle, wherein

the at least one second action includes a plurality of second actions,

the candidate determination unit, in operation, determines the plurality of second actions and determines a priority order of each of the plurality of second actions,

the image generator, in operation, generates a plurality of third images representing the plurality of second actions,

the image output unit, in operation, outputs the plurality of third images to the display in the vehicle to display the plurality of third images in a manner according to priority orders.

17. The driving control device according to claim 16, wherein the image output unit outputs the first image and the second image to the display in the vehicle to display the first image and the second image within a fixed field of view of a driver of the vehicle.

18. A driving assistance method executed by a computer, the method comprising:

acquiring, from an autonomous driving controller that determines an action of a vehicle during autonomous driving of the vehicle, action information indicating a first action that the vehicle is caused to execute;

acquiring, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result;

determining, based on the detection information, at least one second action that the vehicle can be caused to execute after the first action indicated by the action information, the first action and the at least one second action adjusting movement of the vehicle, the at least one second action including a plurality of second actions;

determining a priority order of each of the plurality of second actions;

generating a first image representing the first action indicated by the action information and a second image representing the second action; and

generating a plurality of third images representing the plurality of second actions;

outputting the first image, the second image, and the plurality of third images to a display in the vehicle; and

displaying the plurality of third images in a manner according to priority orders.

19. The driving assistance method according to claim 18, further comprising:

displaying the first image and the second image within a fixed field of view of a driver of the vehicle.