SYSTEM AND METHOD OF CONVERSATIONAL ADJUSTMENT BASED ON USER'S COGNITIVE STATE AND/OR SITUATIONAL STATE

• US 20170162197A1
• Filed: 12/06/2016
• Published: 06/08/2017
• Est. Priority Date: 12/06/2015
• Status: Active Grant

• Current Assignee: Cerence Inc.
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: COHEN, Philip R.
• Granted Patent: US 10,431,215 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

A system and method is provided for adjusting natural language conversations between a human user and a computer based on the human user'"'"'s cognitive state and/or situational state, particularly when the user is operating a vehicle. The system may disengage in conversation with the user (e.g., the driver) or take other actions based on various situational and/or user states. For example, the system may disengage conversation when the system detects that the driving situation is complex (e.g., car merging onto a highway, turning right with multiple pedestrians trying to cross, etc.). The system may (in addition or instead) sense the user'"'"'s cognitive load and disengage conversation based on the cognitive load. The system may alter its personality (e.g. by engaging in mentally non-taxing conversations such as telling jokes based on situational and/or user states.

Claims

1. A computer implemented method of adjusting natural language conversations between a user and a computer based on the user'"'"'s cognitive state and/or situational state while operating a vehicle, the method being implemented in the computer having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer to perform the method, the method comprising:

receiving, by the computer, a first utterance from a user during a conversation with the user;

obtaining, by the computer, sensor information relating to at least a first physiological characteristic of the user;

determining, by the computer, a cognitive state of the user based on the sensor information;

determining, by the computer, whether to disengage from the conversation with the user based on the cognitive state of the user; and

delaying, by the computer, a response to the first utterance responsive to a determination to disengage from the conversation with the user.

2. The method of claim 1, the method further comprising:

obtaining, by the computer, second sensor information relating to at least a second physiological characteristic of the user;

determining, by the computer, a second cognitive state of the user based on the sensor information;

determining, by the computer, that the conversation should be re-engaged with the user based on the second cognitive state of the user; and

re-engaging, by the computer, the conversation with the user, wherein re-engaging includes providing the response to the first utterance.

3. The method of claim 2, the method further comprising:

receiving, by the computer, a second utterance from the user while the computer is disengaged from the conversation with the user; and

adding, by the computer, the second utterance to a conversation stack that includes the first utterance, wherein re-engaging, by the computer, the conversation with the user comprises processing the second utterance to determine its intent in relation to the first utterance.

4. The method of claim 1, the method further comprising:

altering, by the computer, a behavior of the computer, other than disengaging in the conversation with the user, based on the cognitive state.

5. The method of claim 4, wherein altering the behavior comprises:

providing, by the computer, a joke based on the cognitive state.

6. The method of claim 1, the method further comprising:

obtaining, by the computer, information relating to the vehicle or its surroundings; and

determining, by the computer, the situational state based on the information relating to the vehicle or its surroundings,

wherein determining whether to disengage from the conversation with the user is based further on the situational state.

7. The method of claim 1, wherein the information relating to the vehicle or its surroundings comprises information relating to a location or heading indicating heightened driver attention is required.

8. The method of claim 7, wherein the information relating to the location or heading indicating heightened driver attention is required comprises information indicating that the user is entering a highway, is nearby a crosswalk with pedestrians, is nearby a road hazard, or is nearby heavy traffic.

9. The method of claim 6, the method further comprising:

providing, by the computer, a natural language alert to the user based on the cognitive state of the user or based on the situational state.

10. The method of claim 1, wherein the vehicle comprises an automobile and wherein the computer is integrated with the automobile.

11. A system of adjusting natural language conversations between a user and a computer based on the user'"'"'s cognitive state and/or situational state while operating a vehicle, the system comprising:

a computer comprising one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer to;

receive a first utterance from a user during a conversation with the user;

obtain sensor information relating to at least a first physiological characteristic of the user;

determine a cognitive state of the user based on the sensor information;

determine whether to disengage from the conversation with the user based on the cognitive state of the user; and

delay a response to the first utterance responsive to a determination to disengage from the conversation with the user.

12. The system of claim 11, wherein the computer is further programmed to:

obtain second sensor information relating to at least a second physiological characteristic of the user;

determine a second cognitive state of the user based on the sensor information;

determine that the conversation should be re-engaged with the user based on the second cognitive state of the user; and

re-engage the conversation with the user, wherein re-engaging includes providing the response to the first utterance.

13. The system of claim 12, wherein the computer is further programmed to:

receive a second utterance from the user while the computer is disengaged from the conversation with the user; and

add the second utterance to a conversation stack that includes the first utterance, wherein re-engaging, by the computer, the conversation with the user comprises processing the second utterance to determine its intent in relation to the first utterance.

14. The system of claim 11, wherein the computer is further programmed to:

alter a behavior of the computer, other than disengaging in the conversation with the user, based on the cognitive state.

15. The system of claim 14, wherein to alter the behavior, the computer is further programmed to:

provide a joke based on the cognitive state.

16. The system of claim 11, wherein the computer is further programmed to:

obtain information relating to the vehicle or its surroundings; and

determine the situational state based on the information relating to the vehicle or its surroundings,

wherein to determine whether to disengage from the conversation with the user is based further on the situational state.

17. The system of claim 11, wherein the information relating to the vehicle or its surroundings comprises information relating to a location or heading indicating heightened driver attention is required.

18. The system of claim 17, wherein the information relating to the location or heading indicating heightened driver attention is required comprises information indicating that the user is entering a highway, is nearby a crosswalk with pedestrians, is nearby a road hazard, or is nearby heavy traffic.

19. The system of claim 16, wherein the computer is further programmed to:

provide a natural language alert to the user based on the cognitive state of the user or based on the situational state.

20. The system of claim 11, wherein the vehicle comprises an automobile and wherein the computer is integrated with the automobile.