Adaptive interactive voice system

• US 10,137,902 B2
• Filed: 02/12/2015
• Issued: 11/27/2018
• Est. Priority Date: 02/12/2015
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Juneja, Ajay, Marti, Stefan, Di Censo, Davide
• Primary Examiner(s): Ghulamali, Qutbuddin
• Application Number:
  US14/620,742
• Publication Number:
  US 20160236690A1
• Time in Patent Office: 1,384 Days
• Field of Search: 340576
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2040/089 Driver voice
• CPC Class Codes: B60W 2050/0029 Mathematical model of the driver
• CPC Class Codes: B60W 2050/0031 Mathematical model of the vehicle
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2510/0208 Clutch engagement state, e.g. engaged or disengaged
• CPC Class Codes: B60W 2510/0604 Throttle position
• CPC Class Codes: B60W 2510/1005 Transmission ratio engaged
• CPC Class Codes: B60W 2520/10 Longitudinal speed
• CPC Class Codes: B60W 2530/20 Tyre data
• CPC Class Codes: B60W 2540/10 Accelerator pedal position
• CPC Class Codes: B60W 2540/12 Brake pedal position
• CPC Class Codes: B60W 2540/14 Clutch pedal position
• CPC Class Codes: B60W 2540/16 Ratio selector position
• CPC Class Codes: B60W 2540/18 Steering angle
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 : Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2554/00 : Input parameters relating to objects
• CPC Class Codes: B60W 2555/20 : Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2556/50 : of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60Y 2302/09 : Reducing the workload of driver
• CPC Class Codes: G01C 21/3629 : Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G01C 21/3661 : Guidance output on an external device, e.g. car radio
• CPC Class Codes: G06F 3/16 : Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/096872 : where instructions are given per voice
• CPC Class Codes: G08G 1/096894 : where input is assisted by the navigation device, i.e. the user does not type the complete name of the destination, e.g. using zip codes, telephone numbers, progressively selecting from initial letters
• CPC Class Codes: G10L 13/04 : Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 25/63 : for estimating an emotional state

Abstract

In one embodiment, an apparatus for adaptively interacting with a driver via voice interaction is provided. The apparatus includes a computational models block and an adaptive interactive voice system. The computational models block is configured to receive driver related parameters, vehicle related parameters, and vehicle environment parameters from a plurality of sensors. The computational models block is further configured to generate a driver state model based on the driver related parameters and to generate a vehicle state model based on the vehicle related parameter. The computational models block is further configured to generate a vehicle environment state model based on the vehicle environment parameters. The adaptive interactive voice system is configured to generate a voice output based on a driver'"'"'s situation and context as indicated on information included within at least one of the driver state model, the vehicle state model, and the vehicle environment state model.

Claims

1. An apparatus for adaptively interacting with a driver via voice interaction, the apparatus comprising:

a computational models block configured to;

receive driver related parameters, vehicle related parameters, and vehicle environment parameters from a plurality of sensors and employ modeling software with instructions that are executed on one or more devices to;

generate a driver state model based on the driver related parameters,

generate a vehicle state model based on the vehicle related parameter, and

generate a vehicle environment state model based on the vehicle environment parameters; and

an adaptive interactive voice system configured to;

generate a voice output based on a driver'"'"'s situation and context as indicated on information included within at least one of the driver state model, the vehicle state model, and the vehicle environment state model;

determine, based on the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model, that the driver is engaged in a conversation with a person; and

refrain from outputting the voice output while the driver is engaged in the conversation with the person to avoid interrupting the conversation.

2. The apparatus of claim 1 wherein the adaptive interactive voice system is configured to generate the voice output based on the driver'"'"'s situation and context by varying a tone of the voice output based on the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model.

3. The apparatus of claim 1 wherein the adaptive interactive voice system is configured to refrain from generating the voice output to prevent driver distraction based on the information included within the at least one of the driver state model, the vehicle state model and the vehicle environment state model indicating that the driver is experiencing a high cognitive load.

4. The apparatus of claim 1 wherein the driver state model includes at least one of a physiological state of the driver, a cognitive load of the driver, a stress level of the driver, an alertness state of the driver, and an emotional state of the driver.

5. The apparatus of claim 1 wherein the vehicle state model includes at least one of throttle position, clutch position, transmission gear selection, brake pedal status, vehicle speed, and tire condition.

6. The apparatus of claim 1 wherein the vehicle environment state model comprises at least one of weather and objects positioned exterior to the vehicle.

7. The apparatus of claim 1 wherein the adaptive interactive system is further configured to generate the voice output to include various levels of verbosity and formality.

8. The apparatus of claim 1 wherein the plurality of sensors include at least one microphone configured to receive a voice input from a driver and to perform voice tone recognition on the voice input.

9. A computer-program product embodied in a non-transitory computer readable medium that includes instructions that, when executed by a processor, configure the processor to:

receive driver related parameters, vehicle related parameters, and vehicle environment parameters from a plurality of sensors and employ modeling software with instructions that are executed on one or more devices to;

generate a driver state model based on the driver related parameters,

generate a vehicle state model based on the vehicle relate parameters, and

generate a vehicle environment state model based on the vehicle environment parameters;

generate a voice output based on a driver'"'"'s situation and context as indicated on information included within at least one of the driver state model, the vehicle state model, and the vehicle environment state model;

determine, based on the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model, that the driver is engaged in a conversation with a person; and

refrain from outputting the voice output while the driver is engaged in the conversation with the person to avoid interrupting the conversation.

10. The computer product of claim 9 further comprising instructions to provide the voice output based on the driver'"'"'s situation and context by varying a tone of the voice output based on the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model.

11. The computer product of claim 9 further comprising instructions to refrain from providing the voice output to prevent driver distraction in response to the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model indicating that the driver is experiencing a high cognitive load.

12. The computer product of claim 9 wherein the driver state model includes at least one of a physiological state of the driver, a cognitive load of the driver, a stress level of the driver, an alertness state of the driver, and an emotional state of the driver.

13. The computer product of claim 9 wherein the vehicle state model includes at least one of a throttle position, clutch position, transmission gear selection, brake pedal status, vehicle speed and tire condition.

14. The computer product of claim 9 wherein the vehicle environment state model comprises at least one of weather and objects positioned exterior to the vehicle.

15. The computer product of claim 9 further comprising instructions to provide the voice output to include various levels of verbosity and formality.

16. An apparatus for adaptively interacting with a driver via voice interaction, the apparatus comprising:

a computational models block programmed to receive driver related parameters, vehicle related parameters, and vehicle environment parameters from a plurality of sensors and employ modeling software with instructions that are executed on one or more devices to;

generate a driver state model based on the driver related parameters,

generate a vehicle state model based on the vehicle relate parameters, and

generate a vehicle environment state model based on the vehicle environment parameters; and

an adaptive interactive voice system programmed to;

generate a voice output based on a driver'"'"'s situation and context as indicated on information included within at least one of the driver state model, the vehicle state model, and the vehicle environment state model;

determine, based on the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model, that the driver is engaged in a conversation with a person; and

refrain from outputting the voice output while the driver is engaged in the conversation with the person to avoid interrupting the conversation.

17. The apparatus of claim 16 wherein the adaptive interactive voice system is further programmed to provide a voice output based on a driver'"'"'s situation and context as indicated to the information included with the at least one of the driver state model, the vehicle state model, and the vehicle environment state model.

18. The apparatus of claim 17 wherein the adaptive interactive voice system is further programmed to provide a voice output based on the driver'"'"'s situation and context by varying a tone of the voice output based on the information included within the at least one of the driver state model, the vehicle state model, and the vehicle environment state model.

19. The apparatus of claim 18 wherein the driver state model includes at least one of a physiological state of the driver, a cognitive load of the driver, a stress level of the driver, an alertness state of the driver, and an emotional state of the driver.

20. The apparatus of claim 18 wherein the vehicle state model includes at least one of a throttle position, clutch position, transmission gear selection, brake pedal status, vehicle speed and tire condition.