Co-pilot and conversational companion

• US 10,720,156 B2
• Filed: 02/21/2018
• Issued: 07/21/2020
• Est. Priority Date: 02/21/2018
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Friedman, Scott A., Remegio, Prince R., Falkenmayer, Tim Uwe, Kyle, Roger Akira, Kakimi, Ryoma, Heide, Luke D., Puranik, Nishikant Narayan
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US15/901,141
• Publication Number:
  US 20190255995A1
• Time in Patent Office: 881 Days
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 28/02 responsive to conditions relating to the driver
• CPC Class Codes: B60K 28/10 responsive to conditions relating to the vehicle
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/265 Voice
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Methods and systems for communicating with a vehicle companion are disclosed. The vehicle companion can include receiving state information about an operator and environmental information about a vehicular environment. A cognitive load and a cognitive requirement can be determined using the state information and the environmental information, respectively. The cognitive load and the cognitive requirement are then compared to determine if an intervention indicator exists. The vehicle companion can then enter into conversation with the operator in response to the intervention indicator, including topics selected based on the operator'"'"'s personality.

Claims

1. A vehicle companion system for vehicle-to-user interaction, comprising:

one or more processors; and

a memory communicably coupled to the one or more processors and storing;

a cognitive monitoring module including instructions that when executed by the one or more processors cause the one or more processors to receive, using one or more internal sensors, state information about an operator, to receive, using one or more external sensors, environmental information about a vehicular environment, to determine a cognitive load of the operator using the state information, the cognitive load being a measure of the mental effort being exerted by the operator, to determine a cognitive requirement of the vehicular environment using the environmental information, the cognitive requirement being a measure of the attention, alertness, or awareness desired from the operator for vehicle operation within an environment, to compare the cognitive load and the cognitive requirement to determine if an intervention indicator exists, and to determine whether there is an anticipated intervention indicator, for the operator, using the environmental information; and

a conversation module including instructions that when executed by the one or more processors cause the one or more processors to emit natural language speech via an audio output device in response to the intervention indicator or the anticipated intervention indicator, the natural language speech prompting a conversation with the operator, the conversation including a topic selected based on stored personality profile information.

2. The vehicle companion system of claim 1, further comprising a personality profile module including instructions that when executed by the one or more processors cause the one or more processors to receive one or more personality inputs from the operator, and to store the one or more personality inputs as personality profile information in a database.

3. The vehicle companion system of claim 1, wherein the intervention indicator is indicated by the cognitive load being outside of a cognitive load range set by the cognitive requirement.

4. The vehicle companion system of claim 1, wherein the cognitive load is an anticipated cognitive load.

5. The vehicle companion system of claim 1, wherein the cognitive monitoring module further includes instructions to determine a second cognitive load after an operator response, and to compare the second cognitive load to the cognitive requirement to determine if the intervention indicator exists.

6. The vehicle companion system of claim 5, wherein the cognitive monitoring module further includes instructions to compare an adjusted cognitive load to the cognitive load to determine a cognitive load change.

7. A non-transitory computer-readable medium for vehicle-to-user interaction and storing instructions that when executed by one or more processors cause the one or more processors to:

receive, using one or more internal sensors, state information about an operator;

receive, using one or more external sensors, environmental information about a vehicular environment;

determine a cognitive load of the operator using the state information, the cognitive load being a measure of the mental effort being exerted by the operator;

determine a cognitive requirement of the vehicular environment using the environmental information, the cognitive requirement being a measure of the attention, alertness, or awareness desired from the operator for vehicle operation within an environment;

compare the cognitive load and the cognitive requirement to determine if an intervention indicator exists;

determine whether there is an anticipated intervention indicator, for the operator, using the environmental information; and

emit natural language speech via an audio output device in response to the intervention indicator or the anticipated intervention indicator, the natural language speech prompting a conversation with the operator, the conversation including a topic selected based on stored personality profile information.

8. The non-transitory computer-readable medium of claim 7, further including instructions to receive one or more personality inputs from the operator, and to store the one or more personality inputs as personality profile information in a database.

9. The non-transitory computer-readable medium of claim 7, wherein the cognitive load is an anticipated cognitive load.

10. The non-transitory computer-readable medium of claim 7, further including instructions to determine a second cognitive load after an operator response, to compare the second cognitive load to the cognitive requirement to determine if the intervention indicator exists, and to compare an adjusted cognitive load to the cognitive load to determine a cognitive load change.

11. A method for vehicle-to-user interaction, comprising:

receiving, using one or more internal sensors, state information about an operator;

receiving, using one or more external sensors, environmental information about a vehicular environment;

determining a cognitive load of the operator using the state information, the cognitive load being a measure of the mental effort being exerted by the operator;

determining a cognitive requirement of the vehicular environment using the environmental information, the cognitive requirement being a measure of the attention, alertness, or awareness desired from the operator for vehicle operation within an environment;

comparing the cognitive load and the cognitive requirement to determine if an intervention indicator exists;

determining whether there is an anticipated intervention indicator, for the operator, using the environmental information; and

emitting natural language speech via an audio output device in response to the intervention indicator or the anticipated intervention indicator, the natural language speech prompting a conversation with the operator, the conversation including a topic selected based on stored personality profile information.

12. The method of claim 11, further comprising receiving one or more personality inputs from the operator, and storing the one or more personality inputs as personality profile information in a database.

13. The method of claim 11, wherein the cognitive load is an anticipated cognitive load.

14. The method of claim 11, further including instructions to determine a second cognitive load after an operator response, to compare the second cognitive load to the cognitive requirement to determine if the intervention indicator exists, and to compare an adjusted cognitive load to the cognitive load to determine a cognitive load change.