Adaptive infotainment system based on vehicle surrounding and driver mood and/or behavior

• US 10,170,111 B2
• Filed: 02/01/2017
• Issued: 01/01/2019
• Est. Priority Date: 01/19/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Puranik, Nishikant Narayan
• Primary Examiner(s): Pullias, Jesse S
• Application Number:
  US15/422,024
• Publication Number:
  US 20180204570A1
• Time in Patent Office: 699 Days
• Field of Search: 704231-257, 704270-275
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 21/41422 located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/45 Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts

Abstract

A vehicular infotainment system, a vehicle and a method of controlling interaction between a vehicle driver and an infotainment system. A multimedia device, a human-machine interface and sensors are used to collect in-vehicle driver characteristic data and extra-vehicle driving conditions. The system additionally includes—or is otherwise coupled to—a computer to convert one or both of traffic pattern data and vehicular positional data into a driver elevated cognitive load profile. In addition, the computer converts the driver characteristic data into a driver mood profile. The system can process these profiles to selectively adjust one or both of the amount of time needed to accept audio commands from a driver and the amount of time needed to provide an audio response to the driver in situations where the system determines the presence of at least one of the elevated cognitive load and a driver mood.

Claims

1. A vehicular infotainment system comprising:

a multimedia device comprising a human-machine interface and at least one of an audio player and a video player;

a plurality of sensors for acquiring driver characteristic data and at least one of traffic pattern data and vehicular positional data; and

a computer signally cooperative with the sensors and multimedia device and comprising a processor and a memory signally coupled to one another such that program instructions stored in the memory and executable by the processor;

cause the computer to convert the acquired at least one of traffic pattern data and vehicular positional data into a driver elevated cognitive load profile;

cause the computer to convert a first portion of the driver characteristic data that is input into the multimedia device from the group consisting of voice speed, voice volume, voice frequency and facial expression into a baseline driver mood profile;

cause the computer to convert a second portion of the driver characteristic data that is input into the multimedia device from the group consisting of voice speed, voice volume, voice frequency and facial expression into an instant driver mood profile wherein at least one of the second portion of the driver characteristic data is quantitatively different from a respective one of the first portion of the driver characteristic data;

cause the computer to selectively adjust the amount of time that the human-machine interface accepts the second portion of the driver characteristic data from a driver in situations where a determination is made by the computer of at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile; and

cause the computer to selectively adjust at least one of an audio response and a video response from the multimedia device to a driver in situations where a determination is made by the computer of at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile, wherein when the computer selectively adjusts at least one of the audio response and the video response from the multimedia device to a driver, it refrains from providing the human-machine interface response at least until such time as the determination by the computer indicates that the respective at least one of the elevated cognitive load profile and the difference between the instant driver mood profile and the baseline driver mood profile is no longer present.

2. The system of claim 1, wherein the human machine interface comprises at least one of a microphone and a camera with which to acquire the driver characteristic data.

3. The system of claim 1, wherein the quantitatively different driver mood profile corresponds to a mood indicative of driver impatience.

4. The system of claim 1, wherein the plurality of sensors for acquiring at least one of traffic pattern data and vehicular positional data further comprises at least one sensor for acquiring at least one of a braking pattern and accelerating pattern.

5. The system of claim 1, wherein the plurality of sensors for acquiring at least one of traffic pattern data and vehicular positional data is selected from the group consisting of a GPS sensor, a visual sensor, a radar sensor, a lidar sensor and an acoustic sensor.

6. The system of claim 1, wherein the computer is further signally cooperative with a driver calendar such that driver meeting schedule data contained therein is conveyed to the computer to convert such schedule data into an additional parameter for the driver elevated cognitive load profile.

7. The system of claim 1, wherein when the computer selectively adjusts at least one of the audio response and the video response from the multimedia device to a driver, it slows down the human-machine interface response.

8. The system of claim 1, wherein when the computer selectively adjusts at least one of the audio response and the video response from the multimedia device to a driver, it speeds up the human-machine interface response.

9. The system of claim 1, wherein when the computer selectively adjusts at least one of the audio response and the video response from the multimedia device to a driver, it provides a different audio feedback to a driver command than that given in response to a driver command where neither the instant driver mood profile or the elevated cognitive load profile are present.

10. The system of claim 1, wherein the at least one of the audio response and the video response from the multimedia device to a driver is aborted once both the elevated cognitive load profile and the difference between the instant driver mood profile and the baseline driver mood profile is present.

11. A vehicle comprising:

a platform comprising a wheeled chassis defining passenger compartment and an engine compartment comprising an engine disposed therein,

a guidance apparatus cooperative with the wheeled chassis and the engine in order to provide motive control; and

an infotainment system for processing voice input, the infotainment system comprising;

a multimedia device comprising a human-machine interface and at least one of an audio player and a video player;

a plurality of sensors for acquiring driver characteristic data and at least one of traffic pattern data and vehicular positional data; and

a computer signally cooperative with the sensors and multimedia device and comprising a processor and a memory signally coupled to one another such that program instructions stored in the memory and executable by the processor;

cause the computer to convert the acquired at least one of traffic pattern data and vehicular positional data into a driver elevated cognitive load profile;

cause the computer to convert a first portion of the driver characteristic data that is input into the multimedia device from the group consisting of voice speed, voice volume, voice frequency and facial expression into a baseline driver mood profile;

cause the computer to convert a second portion of the driver characteristic data that is input into the multimedia device from the group consisting of voice speed, voice volume, voice frequency and facial expression into an instant driver mood profile wherein at least one of the second portion of the driver characteristic data is quantitatively different from a respective one of the first portion of the driver characteristic data;

cause the computer to selectively adjusts the amount of time that the human-machine interface accepts the second portion of the driver characteristic data from a driver in situations where a determination is made by the computer of at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile; and

cause the computer to selectively adjust at least one of an audio response and a video response from the multimedia device to a driver in situations where a determination is made by the computer of at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile, wherein when the computer selectively adjusts at least one of the audio response and the video response from the multimedia device to a driver, it refrains from providing the human-machine interface response at least until such time as the determination by the computer indicates that the respective at least one of the elevated cognitive load profile and the difference between the instant driver mood profile and the baseline driver mood profile is no longer present.

12. The vehicle of claim 11, wherein the baseline driver mood profile comprises a profile selected from the group consisting of voice speed, voice volume, voice frequency and facial expression, and further wherein the instant driver mood profile comprises a profile selected from the group consisting of voice speed, voice volume, voice frequency and facial expression at least one of which is quantitatively different from a respective baseline voice speed, voice volume, voice frequency and facial expression.

13. A method of controlling interaction between a vehicle driver and an infotainment system, the method comprising:

acquiring driver characteristic data using at least one sensor mounted within a passenger compartment of a vehicle;

configuring the infotainment system to comprise;

a multimedia device with a human-machine interface;

a computer signally cooperative with the multimedia device and the at least one sensor, the computer comprising a processor and a memory signally coupled to one another; and

program instructions stored in the memory and executable by the processor;

using the program instructions to convert a first portion of the driver characteristic data that is input into the multimedia device from the group consisting of voice speed, voice volume, voice frequency and facial expression into a baseline driver mood profile;

using the program instructions to convert a second portion of the driver characteristic data that is input into the multimedia device from the group consisting of voice speed, voice volume, voice frequency and facial expression into an instant driver mood profile wherein at least one of the second portion of the driver characteristic data is quantitatively different from a respective one of the first portion of the driver characteristic data;

acquiring at least one of traffic pattern data and vehicular positional data using at least one vehicle-mounted sensor;

using the program instructions to selectively adjust the amount of time that the human-machine interface accepts the second portion of the driver characteristic data from a driver if a determination is made by the computer of at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile; and

using the program instructions to selectively adjust at least one of an audio response and a video response from the multimedia device to a driver if a determination is made by the computer of at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile, wherein when the computer selectively adjusts at least one of the audio response and the video response from the multimedia device to a driver, it refrains from providing the human-machine interface response at least until such time as the determination by the computer indicates that the respective at least one of the elevated cognitive load profile and the difference between the instant driver mood profile and the baseline driver mood profile is no longer present.

14. The method of claim 13, wherein using the program instructions to selectively lengthen the amount of time that the human-machine interface accepts the second portion of the driver characteristic data from a driver comprises lengthening an end-of-speech timeout.

15. The method of claim 13, wherein using the program instructions to selectively lengthen the amount of time that the human-machine interface accepts the second portion of the driver characteristic data from a driver comprises lengthening a maximum speech timeout.

16. The method of claim 13, wherein using the program instructions to selectively adjust the at least one of the audio response and the video response from the multimedia device to a driver comprises at least one of (a) abbreviating such response relative to when no such at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile is determined, (b) delaying such response relative to when no such at least one of the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile is determined and (c) aborting such response when both the elevated cognitive load profile and a difference between the instant driver mood profile and the baseline driver mood profile is determined.

17. The method of claim 13, wherein using the program instructions to selectively adjust the at least one of the audio response and the video response from the multimedia device to a driver comprises delivering at least one earcon to the driver.